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We  are  providing  this  final  audit  report  for  your  review  and  comments.  It 
discusses  the  use  of  prototyping  in  the  acquisition  strategies  for  major  Defense 
acquisition  programs.  Comments  on  a  draft  of  this  report  were  considered  in  preparing 
the  final  report. 

DoD  Directive  7650.3  requires  that  all  recommendations  be  resolved  promptly. 
Therefore,  we  request  that  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  and  the  Director,  Program  Analysis  and  Evaluation,  provide  comments  on 
the  unresolved  recommendations  and  internal  control  weaknesses  by  November  1, 

1994. 


We  appreciate  the  courtesies  extended  to  our  audit  staff.  If  you  have  questions 
on  this  audit,  please  contact  Mr.  John  Meling,  Program  Director,  at  (703)  604-9090 
(DSN  664-9090)  or  Mr.  Harold  James,  Project  Manager,  at  (703)  604-9093 
(DSN  664-9093).  Appendix  L  lists  the  distribution  of  this  report.  The  audit  team 
members  are  listed  inside  the  back  cover. 

Robert!.  Lieberman 
Assistant  Inspector  General 
for  Auditing 
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Repoi-t  No.  94-181  September  2, 1994 

(Project  No.  2AE-0051) 

THE  EFFECTIVENESS  OF  PROTOTYPING  ACQUISITION 
STRATEGIES  FOR  MAJOR  DEFENSE  ACQUISITION  PROGRAMS 


EXECUTIVE  SUMMARY 


Introduction.  A  prototype  has  traditionally  been  defined  as  a  model  upon  which  a 
later  item  is  formed  or  based  and  which  provides  a  basis  to  test  new  types  of  equipment 
before  large-scale  use  of  resources.  Prototypes  are  developed  and  tested  to  reduce 
technical  risk  in  weapons  system  development  by  providing  information  on  the 
feasibility  of  planned  system  concept  and  design,  as  well  as  expected  operational 
effectiveness  and  suitability.  Prototypes  should  be  used  early  in  the  acquisition 
process,  with  a  prototyping  strategy  established  at  the  Milestone  I,  Concept 
Demonstration  Approval,  and  executed  during  Phase  I,  Demonstration  and  Validation. 
Use  of  prototypes  allows  the  Government  to  delay  major  funding  commitments  until 
system  design  is  more  advanced  and  risk  is  reduced.  A  prototype  is  not  normally  a 
complete  system  but  is  designed  to  focus  on  the  areas  of  highest  technical  risk. 
Numerous  reviews  of  the  DoD  acquisition  process  have  advocated  prototyping. 

Objectives.  The  primary  objective  of  the  audit  was  to  evaluate  the  effectiveness  of 
DoD's  use  of  prototyping  acquisition  strategies  for  major  Defense  acquisition  systems 
and  critical  subsystems.  We  also  evaluated  the  internal  controls  associated  with  the 
prototyping  process. 

Audit  Results.  The  use  of  prototyping  for  major  Defense  acquisition  programs  has 
increased  over  the  last  7  years.  Noteworthy  examples  of  prototyping  included  the 
Upper  Tier  Theater  Missile  Defense  System  and  the  Air  Force  F-22  Aircraft  Program. 
However,  further  improvement  was  needed.  Prototyping  could  be  used  more 
effectively  in  the  acquisition  strategies  for  major  Defense  acquisition  programs  to 
identify  and  reduce  risk  and  to  assess  whether  the  most  promising  design  approaches 
will  operate  in  the  intended  operational  environment  before  proceeding  with  Phase  E, 
Engineering  and  Manufacturing  Development.  The  effectiveness  of  prototyping 
acquisition  strategies  for  five  of  the  six  programs  reviewed  was  impeded  by  a  lack  of  a 
properly  structured  planning  and  decisionmaking  process  for  balancing  cost,  schedule, 
and  technical  risk  and  by  noncompliance  with  existing  regulations.  As  a  result,  DoD 
incurred  increased  risk  of  committing  system  designs  to  engineering  and  manufacturing 
development  when  feasibility  of  the  designs  and  their  ability  to  meet  mission  needs  had 
not  been  sufficiently  demonstrated. 

Internal  Controls.  Internal  controls  and  the  implementation  of  the  DoD  Internal 
Management  Control  Program  were  not  effective  to  identify  material  internal  control 
weaknesses.  Office  of  the  Secretary  of  Defense  and  Military  Department  directives  and 
instructions  provided  only  limited  guidance  regarding  the  use  of  prototyping.  Guidance 
was  needed  for  formulation  and  execution  of  effective  prototyping  strategies.  These 
internal  control  weaknesses  are  discussed  in  Part  I  of  this  report. 

Potential  Benefits  of  the  Audit.  Implementing  the  recommendations  in  this  report 
will  ensure  that  programs  progress  to  Phase  II,  Engineering  and  Manufacturing 
Development,  only  when  prototyping  and  demonstration  processes  provide  reasonable 


assurance  that  the  technologies  and  processes  critical  to  success  are  attainable  and 
affordable.  Monetary  benefits  to  be  realized  by  implementing  the  recommendations 
should  occur  from  reduced  future  program  growth.  However,  the  benefits  were  not 
readily  quantifiable  since  they  impact  future  programs  transitioning  from  Phase  1, 
Demonstration  and  Validation,  to  Phase  II,  Engineering  and  Manufacturing 
Development.  Appendix  J  notes  the  potential  benefits  to  be  derived  from  implementing 
the  recommendations. 

Summary  of  Recommendations.  We  recommend  that  the  Under  Secretary  of  Defense 
for  Acquisition  and  Technology  improve  the  planning  process  for  use  of  prototyping  in 
acquisition  strategies  and  the  management  of  prototyping  programs  to  ensure  readiness 
for  the  transition  into  Engineering  and  Manufacturing  Development.  We  also 
recommend  that  the  Director,  Program  Analysis  and  Evaluation,  determine  whether 
proposed  prototyping  plans  will  adequately  support  analyses  of  operational  efficiency 
and  costs. 

Management  Comments.  The  Director,  Acquisition  Program  Integration,  provided 
management  comments  to  the  draft  report  for  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology.  The  Director  concurred  with  our  finding  ljut 
nonconcurred  with  the  majority  of  our  recommendations  involving  revisions  of  existing 
policy  related  to  the  use  of  prototyping.  The  Director,  Program  Analysis  and 
Evaluation,  nonconcurred  with  the  finding  but  did  not  provide  specific  comments  to  our 
recommendation  for  him.  The  Director,  Defense  Procurement,  nonconcurred  with  our 
recommended  revision  of  the  Defense  Federal  Acquisition  Regulation  Supplement  to 
provide  for  contractor  input  to  prototyping  plans.  However,  she  proposed  alternative 
corrective  actions  that  would  require  implementation  by  the  Under  Secretary  of  Defense 
for  Acquisition  and  Technology.  Part  II  contains  a  discussion  of  management's 
comments  to  the  report.  Part  IV  contains  the  complete  texts  of  comments  from  the 
Director,  Acquisition  Program  Integration;  the  Director,  Program  Analysis  and 
Evaluation;  and  the  Director,  Defense  Procurement. 

Audit  Response.  We  revised  five  recommendations  in  response  to  comments  provided 
by  the  Director,  Acquisition  Program  Integration.  Additionally,  we  revised  and 
redirected  a  recommendation  concerning  contractor  input  to  proposed  prototyping  plans 
to  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology  rather  than  the 
Director,  Defense  Procurement,  based  on  management  comments.  We  stand  by  all 
other  recommendations  as  written.  We  believe  the  general  comments  provided  by  both 
the  Director,  Acquisition  Program  Integration,  and  the  Director,  Program  Analysis  and 
Evaluation,  as  well  as  the  comments  to  many  individual  recommendations  show  a 
misunderstanding  of  both  the  content  and  intent  of  our  draft  report.  In  our  audit 
responses,  we  provided  additional  information  and  clarification.  We  request  that  the 
Under  Secretary  of  Defense  for  Acquisition  and  Technology  and  the  Director,  Program 
Analysis  and  Evaluation,  reconsider  their  positions  on  the  nonconcurrences  and  provide 
additional  comments  to  the  final  report  by  November  1,  1994. 
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Part  I  -  Introduction 


Background 


Prototyping.  A  prototype  has  traditionally  been  defined  as  a  model  on  which  a 
later  item  is  formed  or  based  that  is  used  to  test  new  types  of  equipment. 
Prototyping  helps  to  reduce  technical  risk  in  developing  weapon  systems  by 
answering  three  questions: 

o  Is  the  concept  feasible? 

o  Does  the  design  work  the  way  it  is  supposed  to  work? 

o  Does  the  system  provide  a  useful  military  capability? 

A  prototype  is  not  normally  a  complete  system  in  the  sense  of  being  deployable 
to  operational  forces.  Rather,  prototypes  should  be  designed  to  focus  on  the 
areas  of  highest  risk  that  are  essential  to  mission  success. 

While  prototyping  requires  initial  time  and  money,  the  Government  can  delay 
major  funding  commitments  until  design  and  development  are  more  advanced 
and  risk  is  reduced.  In  recent  years,  prototyping  has  received  increased 
attention  due  to  decreasing  DoD  budgets. 

Acquisition  Regulations.  DoD  Instruction  5000.2,  "Defense  Acquisition 
Policies  and  Procedures,"  February  23,  1991,  identifies  prototyping  as  a  major 
element  of  Phase  I,  Demonstration  and  Validation,  to  use  in  assessing  and 
reducing  the  risks  from  integrating  available  and  emerging  technologies  to 
satisfy  a  valid  mission.  The  Instruction  provides  the  following  guidance: 

o  Requirements  for  prototyping  are  established  at  Milestone  I, 

Concept  Demonstration  Approval. 

o  Prototyped  technologies  include  hardware,  software,  and 

manufacturing  processes. 

0  Prototyping  is  differentiated  from  the  advanced  technology 
demonstrations  that  may  occur  before  Milestone  I,  Concept  Approval,  on  a 
major  Defense  acquisition  program  (MDAP): 

o  Advanced  technology  demonstrations  assess  the  military  utility 
of  innovative  technologies  and  focus  on  validating  the  viability,  utility,  and 
producibility  of  a  technology. 

o  Prototyping  involves  the  development  and  demonstration  of  a 
system  or  a  subsystem  or  component  of  that  system  to  meet  a  specified  mission 
requirement. 
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o  DoD  Instruction  5000.2  requires  competitive  prototyping  of  systems 
or  critical  subsystems  unless  the  milestone  decision  authority  issues  a  specific 
waiver. 


Objectives 


Our  overall  objective  was  to  evaluate  the  effectiveness  of  DoD  use  of 
prototyping  acquisition  strategies  for  major  Defense  acquisition  systems  and 
critical  subsystems.  We  also  evaluated  the  internal  controls  associated  with 
prototyping.  The  audit  is  one  of  a  series  of  reviews  to  assess  implementation  of 
recommendations  in  the  Packard  Commission  Report  that  the  Secretary  of 
Defense  submitted  to  the  President  in  June  1986. 


Scope  and  Methodology 


To  satisfy  our  objectives,  we  evaluated  the  use  of  prototyping  for  six  MDAPs 
that  were  in  either  in  Phase  I,  Demonstration  and  Validation,  or  had  completed 
Demonstration  and  Validation  and  began  Phase  II,  Engineering  and 
Manufacturing  Development  (EMD),  since  September  1989.  Those  programs 
were  judgmentally  selected  from  the  universe  of  programs  meeting  the  criteria 
for  MDAPs  established  in  DoD  Instruction  5000.2,  "Defense  Acquisition 
Management  Policies  and  Procedures."  Programs  were  selected  from  the  Office 
of  the  Secretary  of  Defense  (OSD)  as  well  from  the  Army,  Navy,  and  Air 
Force: 

o  OSD: 

Upper  Tier  Theater  Missile  Defense  System  (UTTMDS) 
o  Army: 

Comanche  Helicopter 

Brilliant  Anti-Armor  Submunition  (BAT) 
o  Navy: 

Joint  Standoff  Weapon  System  (JSOW) 

Fixed  Distributed  System  (FDS) 
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o  Air  Force: 

F-22  Aircraft 

We  also  performed  limited  examinations  of  three  other  programs:  Army 
Tactical  Missile,  Navy  F/A-18  E/F  Aircraft,  and  Air  Force  Joint  Surveillance 
and  Target  Attack  Radar  System.  Audit  work  on  the  Army  Tactical  Missile  and 
the  Joint  Surveillance  and  Target  Attack  Radar  System,  both  of  which  had 
began  Phase  II,  EMD,  by  February  1986,  only  evaluated  the  role  of  the 
Advanced  Research  Projects  Agency  (ARP A)  in  prototyping.  Those  progr^s 
were  selected  because  they  were  latest  examples  of  ARPA  involvement  in  a 
MDAP  during  Phase  I,  Demonstration  and  Validation.  Audit  work  on  the 
F/A-18  E/F  Aircraft  program  was  limited  to  a  review  that  validated  the  Navy’s 
decision  not  to  use  prototypes  because  the  program  was  not  high  risk  in  terms  of 
concept,  design,  performance,  or  operational  suitability. 

We  examined  information  in  acquisition  plans,  operational  requirements 
documents,  contract  files,  test  and  evaluation  master  plans,  developmental  and 
operational  test  assessments  and  reports,  and  technical  reviews  and  audits 
performed  to  support  start  and  progression  through  Phase  I,  Demonstration  and 
Validation,  and  transition  to  Phase  II,  EMD.  We  interviewed  personnel 
responsible  for  program  management,  procurement,  testing,  and  contract 
administration,  as  well  as  contractor  personnel,  to  determine  and  evaluate  the 
policies  and  procedures  followed  in  the  prototyping  process.  We  examined  the 
program  information  provided  to  senior  acquisition  management  in  the  Defense 
Acquisition  Executive  Summary  Report  and  other  reports.  We  also  assessed 
support  for  waiving  programs  from  the  requirements  for  competitive 
prototyping,  risk  reduction  efforts  in  programs  where  prototyping  was  not  used 
or  planned,  and  use  of  test  and  evaluation  of  prototypes  to  support  program 
transition  to  Phase  II,  EMD. 

As  a  supplement  to  our  audit  coverage,  we  surveyed  15  program  offices  for 
MDAPs  that  started  after  January  1987  to  determine  the  extent  to  which 
prototyping  has  been  planned  or  used  since  the  issuance  of  the  Packard 
Commission  Report  in  1986  (Appendix  B).  To  obtain  industry  perspectives  on 
use  of  prototypes,  we  surveyed  six  contractors  who  had  participated  as  prime 
contractors  in  the  development  of  prototypes  of  MDAPs  during  Phase  I, 
Demonstration  and  Validation.  We  obtained  their  input  concerning  how 
prototyping  can  be  used  most  effectively  (Appendix  C). 

The  audit  began  in  June  1992  and  was  suspended  in  October  1992.  Audit  work 
resumed  in  March  1993  and  was  complete  in  February  1994.  The  audit  work 
included  a  review  of  information  relating  to  prototyping  dated  from 
December  1984  through  February  1994.  This  program  results  audit  was  made 
in  accordance  with  auditing  standards  issued  by  the  Comptroller  General  of  the 
United  States,  as  implemented  by  the  Inspector  General,  DoD,  and  included 
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necessary  tests  of  internal  controls.  We  did  not  place  material  reliance  on 
computer-processed  data  to  support  the  finding  and  recommendations  in  this 
audit  report.  Appendix  K  lists  the  organizations  visited  or  contacted. 


Internal  Controls 


We  assessed  the  internal  controls  related  to  the  prototyping  process.  Those 
controls  are  specified  in  DoD  Directive  5000.1,  "Defense  Acquisition," 
February  23,  1991,  and  DoD  Instruction  5000.2,  "Defense  Acquisition 
Management  Policies  and  Procedures,"  February  23,  1991.  We  also  assessed 
implementation  of  DoD  Directive  5010.38,  "DoD  Internal  Management  Control 
Program,"  April  14,  1987.  As  part  of  our  evaluation,  we  assessed: 

o  Statutory  and  DoD  regulatory  guidance  on  the  use  prototyping; 

o  Military  Department  implementing  procedures  and  compliance  with 
regulations;  and 

o  OSD  and  DoD  Component  oversight  of  the  use  of  prototypes  in 
program  acquisition  strategies. 

The  audit  identified  material  internal  control  weaknesses,  as  defined  by  DoD 
Directive  5010.38,  "Internal  Management  Control  Program,"  April  14,  1987. 
Guidance  for  the  prototyping  process  needed  improvement  to  ensure  that 
prototypes  were  used  more  effectively  in  reducing  technical,  cost,  and  schedule 
risk  when  transitioning  from  Phase  I,  Demonstration  and  Validation,  to  Phase 
II,  EMD.  The  Internal  Management  Control  Program  failed  to  detect  the 
internal  control  weakness  because  it  did  not  include  the  prototyping  process  as 
an  assessable  unit.  Moreover,  the  management  comments  on  the  draft  report 
indicate  that  management  would  not  have  considered  the  process  to  have 
internal  control  weaknesses  even  if  a  vulnerability  assessment  had  been  done. 

Implementation  of  the  recommendations  in  this  report  will  correct  those 
weaknesses.  The  monetary  benefits  of  implementing  those  recommendations 
were  undeterminable  because  the  recommendations  will  impact  future  programs 
entering  Phase  I,  Demonstration  and  Validation,  and  transitioning  to  Phase  II, 
EMD.  A  copy  of  the  report  will  be  provided  to  the  senior  officials  responsible 
for  internal  controls  within  the  Offices  of  the  Secretary  of  Defense  and  the 
Military  Departments. 
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Prior  Audits  and  Other  Reviews 


Since  1989,  the  Inspector  General,  DoD,  and  the  General  Accounting  Office 
have  each  issued  three  reports  addressing  issues  that  relate  to  the  use  of 
prototyping.  We  identified  six  other  reviews  perform^  by  Federally  Funded 
Research  and  Development  Centers  and  within  DoD  that  made 
recommendations  involving  prototyping  policies  and  procedures.  Appendix  D 
synopsizes  those  audits  and  other  reviews. 
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Part  II  -  Finding  and  Reconunendations 


Use  of  Prototyping  in  Acquisition 
Strategies 

Prototyping  had  not  been  used  effectivelj^  in  the  acquisition  strategies  for 
major  Defense  acquisition  programs  to  identify  and  reduce  risk  and  to 
assess  whether  the  most  promising  design  approaches  will  operate  in  the 
intended  operational  environment  before  proceeding  with  EMD.  The 
effectiveness  of  prototyping  was  impeded  because  a  properly  structured 
acquisition  planning  and  decisionmaking  process  for  balancing  cost, 
schedule,  and  technical  risks  at  Milestone  I,  Concept  Development 
Approval,  did  not  exist  to  determine  how  prototypes  should  be  used 
during  Phase'  I,  Demonstration  and  Validation.  Effectiveness  of 
prototyping  was  also  impeded  by  noncompliance  with  existing 
regulations  and  guidance.  As  a  result,  the  Government  incurred  the 
increased  risk  of  committing  to  EMD  before  system  designs,  including 
critical  technologies  and  manufacturing  processes,  have  sufficiently 
demonstrated  the  potential  to  affordably  satisfy  mission  needs  and 
achieve  acceptable  operational  performance. 


Background 


Use  of  prototyping  is  a  critical  element  in  meeting  the  objectives  provided  by 
DoD  Instruction  5000.2  for  Phase  I,  Demonstration  and  Validation,  of  the 
acquisition  process  for  MDAPs.  Those  objectives  include  supporting  Milestone 
II,  Development  Approval,  through: 

o  Better  defining  the  critical  design  characteristics  and  expected 
capabilities  of  the  system  concept; 

o  Demonstrating  that  critical  technologies  can  be  incorporated  into 
system  designs  with  confidence;  and 

o  Proving  the  processes  critical  to  the  most  promising  system  concepts 
are  understood  and  attainable. 

DoD  Instruction  5000.2  further  provides  that  a  program  may  not  enter  Phase  II, 
EMD,  unless  the  milestone  decision  authority  confirms,  before  the  Milestone  II, 
Development  Approval,  that  prototyping  and  demonstration  results  to  date 
provide  reasonable  assurance  that  the  technologies  and  processes  critical  to 
success  are  attainable.  DoD  Instruction  5000.2  requires  that  hardware, 
software,  and  manufacturing  processes  be  included  within  the  scope  of 
prototyping  efforts. 

One  of  the  most  difficult  aspects  of  planning  a  prototyping  acquisition  strategy 
is  determining  what  should  be  prototyped.  While  full  system  or  process 
prototypes  are  not  warranted  in  all  cases,  some  level  of  prototyping  would  be 
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beneficial  in  every  MDAP  requiring  development  through  the  Demonstration 
and  Validation  and  EMD  phases,  especially  if  electronic  simulation  and  design 
analysis  is  considered  part  of  the  total  prototyping  effort. 

The  results  of  prototyping  efforts  impact  virtually  every  required  document  and 
analysis  specified  to  support  a  Milestone  II,  Development  Approval,  decision. 
For  example,  the  Cost  and  Operational  Effectiveness  Analysis,  one  cornerstone 
to  acquisition  milestone  decisions,  should  be  h^vily  dependent  on  prototyping 
results  to  compare  various  alternatives  for  meeting  military  requirements.  DoD 
Instruction  5000.2  r^uires  that  Cost  and  Operational  Effectiveness  Analysis  be 
prepared  and  considered  at  milestone  decision  reviews  of  major  Defense 
acquisition  programs,  beginning  with  the  Milestone  I,  Concept  Demonstration 
Approval.  At  the  Milestone  II,  Development  Approval,  the  Instruction  states 
that  the  Analysis  should  typically  establish  performance  floor  and  cost  ceiling 
objective  or  acceptable  bands  for  possible  combinations  of  cost  and 
performance.  Performance  of  a  Cost  and  Operational  Effectiveness  Analysis 
without  the  benefit  of  prototyping  results  unduly  restricts  the  analysis  to  stated 
versus  demonstrated  system  capabilities.  Knowledge  gained  through  use  of 
prototyping  increases  the  realism  and,  thus,  the  usefulness  of  the  Analysis.  The 
useMness  of  other  documentation  required  by  DoD  Instruction  5000.2  to 
support  Milestone  II,  such  as  the  Developmental  Test  and  Evaluation  Report, 
the  Early  Operational  Assessment,  and  Program  Life-Cycle  Cost  Estimate 
(program  manager's  estimate  of  the  total  cost  to  the  Government  for  acquisition 
and  ownership  of  a  system  over  its  useful  life)  would  also  be  enhanced  by 
prototyping  experience. 

Prototyping  efforts  require  a  careful  balance  of  cost,  schedule,  and  technical 
factors.  The  knowledge  and  understanding  gained  through  prototyping 
experience  during  Demonstration  and  Validation  provides  the  foundation 
necessary  for  a  program  to  successfully  progress  through  EMD  and 
subsequently  to  meet  prerequisites  for  transitioning  into  production  provided  by 
DoD  Instruction  5000.2;  Military  Standard  152 IB,  "Technical  Reviews  and 
Audits  for  Systems,  Equipments,  and  Computer  Software";  DoD  Standard 
2167A,  "Defense  System  Software  Development";  and  DoD  4245 -7-M, 
"Transition  from  Development  to  Production."  Appendix  E  shows  how  those 
guidance  documents  support  the  transition  from  development  to  production. 
Additional  recommended  prerequisites  are  discussed  in  Inspector  General,  DoD, 
Report  No.  94-014,  "Low-Rate  Initial  Production  in  Major  Defense  Acquisition 
Programs,"  November  9,  1993.  Because  of  the  relevance  of  the  Low-Rate 
Initial  Production  Report,  the  report  recommendations  are  listed  in  Appendix  D. 
The  recommendations  provided  by  Report  No.  94-014  and  this  draft  report, 
along  with  existing  policies  and  procedures,  constitute  a  defined  process  to  plan 
and  execute  risk-reduction  efforts,  including  the  use  of  prototyping,  as  programs 
progress  from  Milestone  I,  Concept  Demonstration  Approval,  through  the  end 
of  low-rate  initial  production.  Appendix  F  shows  the  impact  of  the 
recommendations  in  these  two  reports  on  the  DoD  acquisition  process  as  defined 
by  DoD  Instruction  5000.2. 
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Planning  and  Executing  Prototyping  Strategies 


DoD  could  improve  the  planning  and  execution  of  prototyping  acquisition 
strategies  to  assess  and  reduce  program  risk.  Shortfalls  in  prototyping  strategies 
criticS  to  support  entry  into  EMD  included: 

o  Prototyping  of  hardware,  software,  or  manufacturing  processes  not 
adequately  planned  or  executed  to  reduce  risk  and  provide  opportunity  for  early 
operational  assessment. 

o  Inadequate  provision  for  competitive  design,  manufacture,  and 
testing  of  prototype  systems  under  Demonstration  and  Validation  contracts. 

o  Insufficient  early  operational  assessments  of  prototyping  efforts  to 
confirm  the  feasibility  of  a  design  approach  to  satisfy  mission  need  and  meet 
minimum  acceptable  operational  performance  requirements. 

o  No  significant  program  events  relating  to  the  use  of  prototypes  in 
acquisition  program  baseline  parameters  for  measuring  program  progress  during 
Demonstration  and  Validation. 

Appendix  G  summarizes  deficiencies  relating  to  the  planning  and  execution  of 
prototyping  strategies  in  the  six  MDAPs  reviewed. 

Prototyping  of  Hardware,  Software,  and  Manufacturing  Processes.  Our 
review  of  prototyping  strategies  and  related  documentation  required  by  DoD 
Instruction  5000.2  and  DoD  Standard  2167A,  "Defense  System  Software 
Development,"  February  29,  1988,  showed  that  programs  had  deficiencies  in 
planning  and  executing  prototyping  of  hardware,  software,  and  manufacturing 
processes.  Those  deficiencies  did  or  will  significantly  reduce  the  quantity  and 
quality  of  information  available  to  program  managers  for  identifying  and 
addressing  technical  problems  before  entry  into  EMD  and  for  supporting 
Milestone  II,  Development  Approval. 

Hardware.  Four  of  the  six  programs  reviewed,  including  the  Army 
BAT  and  Comanche  programs  and  the  Navy  FDS  and  JSOW  programs,  had 
limitations  in  planning  or  execution  of  hardware  prototyping  strategies. 

Army  BAT  Program.  The  Army  BAT  Program  provides  an 
example  of  the  programs  having  limitations  in  planning  or  execution  of 
hardware  prototyping  strategies.  The  BAT  program  had  no  formal  Milestone  I, 
Concept  Demonstration  Approval.  In  lieu  of  a  formal  Milestone  I  review,  the 
program  had  a  "Blue  Ribbon  Panel"  review  in  December  1984  that  included 
technical  and  acquisition  experts  from  Government  and  academia.  The  "Blue 
Ribbon  Panel"  provided  a  recommended  acquisition  strategy  for  the  program  to 
follow  to  the  Milestone  II,  Development  Approval,  decision  point.  The  BAT 
Program  Office  followed  the  guidance  provided  by  the  Blue  Ribbon  Panel  with 
one  major  exception.  The  Panel  had  stated  that,  before  the  program  progressed 
to  Milestone  II,  the  complete  prototype  BAT  vehicle  should  be  drop  tested. 
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Drop  testing  involves  release  of  the  BAT  from  an  aircraft  to  determine  whether 
it  can  glide  to  an  assigned  target.  Although  drop  testing  did  occur  before 
Milestone  II,  the  BAT  vehicle  used  had  the  following  limitations  relating  to  key 
subsystems: 


o  no  integrated  Central  Electronics  Unit, 

o  no  integrated  warhead,  and 

o  incomplete  Inertial  Measurement  Unit. 

The  Central  Electronics  Unit  is  composed  of  an  autonomous  processor,  auto¬ 
pilot,  and  interface  electronics.  According  to  program  office  personnel,  the 
Central  Electronics  Unit  was  not  integrated  into  the  BAT  because  of  delays 
experienced  by  the  contractor  in  completing  a  design  that  could  be 
accommodated  within  the  available  space.  The  space  limitations  resulted  from  a 
change  to  the  Required  Operational  Capabilities  on  December  8,  1987,  which 
led  to  a  modification  to  the  warhead  that  increased  the  space  that  it  required, 
thus  limiting  the  space  available  for  other  subsystems.  During  drop  testing, 
functions  to  be  performed  by  the  Central  Electronics  Unit  were  done  through 
telemetry  using  a  receiver  aboard  the  BAT  and  a  ground-based  Central 
Electronics  Unit  breadboard  system. 

The  warhead  or  munition  is  composed  of  a  precursor  charge,  main  charge,  and 
an  electronic  safe  and  arm  device.  Although  test  firings  of  the  warhead  were 
completed  before  Milestone  II,  it  was  not  integrated  into  the  BAT  submunition 
and  used  as  part  of  the  drop  testing  exercises.  Development  of  the  warhead,  as 
discussed  above,  was  impacted  by  the  need  for  modifications  during 
Demonstration  and  Validation  to  increase  its  capabilities. 

The  Inertial  Measurement  Unit  is  part  of  the  guidance  system  for  the  BAT  that 
is  used  to  measure  movement  and  acceleration.  Like  the  Central  Electronics 
Unit,  the  design  of  this  measurement  unit  lagged  behind  due  to  packaging 
difficulties  resulting  from  the  increased  space  required  for  the  warhead.  As  a 
result,  only  a  partial  Inertial  Measurement  Unit  capability  was  available  for 
testing  before  Milestone  II,  Development  Approval. 

Limitations  in  the  availability  of  BAT  prototype  hardware  for  testing  during 
Demonstration  and  Validation  increased  the  level  of  developmental  risk  carried 
over  into  EMD  in  May  1991.  A  briefing  prepared  September  20,  1993,  by  the 
BAT  Program  Office  personnel  for  our  audit  team  stated  that  the  major  risk 
areas  for  Phase  II,  EMD,  included  design,  testing,  and  development  of  the 
Central  Electronics  Unit,  subsystem  integration,  and  effectiveness  of  terminal 
homing.  Progressing  to  EMD  when  risk-reduction  strategy  for  Demonstration 
and  Validation  was  incomplete  is  an  example  of  a  schedule-driven  versus  event- 
driven  acquisition  strategy. 
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Navy  JSOW  Program.  The  Navy  JSOW  Program  provides 
another  example  of  a  system  that  had  significant  limitations  in  the  planning  and 
execution  of  prototyping.  The  statement  of  work  for  contract  N00019-89-C- 
0076,  issued  to  Texas  Instruments  on  June  29,  1989,  to  support  Demonstration 
and  Validation  for  the  program,  did  not  require  the  design,  fabrication,  and 
testing  of  prototype  hardware.  Required  risk  reduction  under  this  statement  of 
work  was  limited  to  wind  tunnel  testing  and  various  systems  engineering 
activities  that  involved  analyses,  modeling,  and  simulation  rather  than  hardware 
prototyping.  Contractors  were  "encouraged"  to  use  prototype  hardware  in 
testing  but  this  effort  was  not  funded  under  the  original  contract.  The  same 
statement  of  work  was  included  in  Demonstration  and  Validation  contracts 
issued  to  two  additional  contractors  who  competed  with  Texas  Instruments 
during  Demonstration  and  Validation. 

When  USD(A&T)  approved  the  JSOW  acquisition  strategy  for  Demonstration 
and  Validation,  USD(A&T)  assumed  that  a  Recoverable  Test  Vehicle  being 
developed  in  a  separate  Navy  program  would  be  used  to  support  captive  flight 
testing  of  airframe  hardware  approximating  the  planned  JSOW  configuration. 
This  test  support  never  occurred  because  the  Recoverable  Test  Vehicle  Program 
encountered  technical  and  financial  difficulties  and  the  vehicle  was  not  available 
to  support  the  JSOW  effort.  Subsequently,  the  competing  contractors  each 
decided  to  pursue  prototyping  activities  outside  of  the  contracts.  The  JSOW 
Program  Office  then  modifi^  the  Demonstration  and  Validation  contracts  in 
September  1990  to  provide  for  receipt  of  test  data  from  the  contractors,  allow 
use  of  Government  facilities  for  testing  dispenser  (launch  and  release 
mechanisms),  and  fund  testing  and  analysis  of  composite  materials  for  the  nose 
and  fins  of  the  JSOW  vehicle. 

Although  the  competing  contractors  built  and  flew  JSOW  prototypes, 
performing  the  vast  majority  of  prototyping  activity  outside  of  the  scope  of  the 
contract  seriously  impacted  the  risk  reduction  accomplishments  that  could  be 
carried  over  into  Phase  II,  EMD.  Factors  contributing  to  the  negative  impact 
on  risk  reduction  included: 

o  Greatly  reduced  opportunity  for  interface  between  the 
JSOW  Program  Office  and  the  contractors  concerning  how  prototyping  could  be 
used  most  effectively  and  efficiently.  Management  personnel  at  all 
three  competing  contractors  agreed  that,  to  be  most  effective  as  a  risk  reduction 
tool,  prototyping  should  be  planned  and  funded  before  beginning  Demonstration 
and  Validation.  Personnel  at  two  of  the  three  contractors  stated,  without 
preliminary  planning  and  funding,  prototyping  becomes  more  of  a  m^ia  event 
than  a  risk-reduction  exercise.  They  stated  that  prototype  hardware  produced 
under  those  conditions  often  has  little  resemblance  to  what  will  be  developed 
during  EMD. 


o  The  lack  of  test  approval  authority  by  the  JSOW 
Program  Office  over  contractor  testing  of  prototype  hardware.  Since  the  testing 
was  done  outside  the  scope  the  awarded  contracts,  the  Program  Office 
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could  not  provide  input  to  the  focus  of  the  testing  process  and  could  not  ensure 
that  the  title  10,  United  States  Code,  section  2438  requirement  for  competitive 
"side  by  side"  prototyping  was  met. 

o  The  lack  of  exit  criteria  involving  events  or 
achievements  using  prototypes  to  support  Milestone  II,  Development  Approval, 
and  award  of  the  EMD  contract.  Exit  criteria  directed  by  USD(A&T)  and 
required  by  DoD  5000.2  were  a  means  to  reduce  program  risk  by  tying 
progression  to  the  next  acquisition  phase  to  specific  events  or  achievements.  All 
exit  criteria  established  in  the  Milestone  I,  Acquisition  Decision  Memorandum, 
June  29,  1988,  were  of  an  administrative  nature  such  as  establishing  a 
Memorandum  of  Understanding  with  the  Air  Force  concerning  joint  Service 
interoperability  and  cooperation  and  did  not  involve  performance-related 
accomplishments. 

Program  Office  personnel  stated  that  an  extra  2  years  were  needed  to  get 
fund^ing  and  negotiate  contract  modifications  and  the  program  Demonstration 
and  Validation  would  cost  an  additional  $34  million  if  the  contracts  had  been 
modified  to  require  competitive  prototyping  for  JSOW.  This  position  does  not 
consider  the  benefits  of  system  and  process  demonstration  in  support  of  source 
selection  and  that  limiting  effort  during  Demonstration  and  Validation  means 
additional  effort  will  be  required  in  EMD.  Additionally,  we  believe  that  the 
time  impact  would  have  been  reduced  if  prototyping  had  been  planned  and  made 
part  of  the  original  Demonstration  and  Validation  contracts  in  June  1989. 

Software.  For  two  of  the  six  programs,  the  Navy  JSOW  and  FDS,  we 
found  significant  limitations  in  the  effectiveness  of  software  prototyping  done 
during  Demonstration  and  Validation.  Additionally,  planning  for  the  test  and 
evaluation  of  system  software  had  been  insufficiently  documented  as  three  of  the 
six  programs.  Army  BAT,  Navy  JSOW,  and  Air  Force  F-22  Aircraft, 
transitioned  from  Demonstration  and  Validation  into  EMD. 

Effectiveness  of  Software  Prototyping.  The  effectiveness  of 
software  prototyping  for  the  JSOW  was  limited  because,  as  discussed  earlier 
under  "hardware,"  the  contractors  did  the  majority  of  prototyping  effort  outside 
the  scope  of  the  Demonstration  and  Validation  contracts,  limiting  planning 
interface  between  the  JSOW  Program  Office  and  the  contractor  and  precluding 
test  approval  authority.  Effectiveness  of  software  prototyping  efforts  were 
further  limited  because  a  thorough  allocation  of  the  software  requirements 
between  the  JSOW  and  the  F/A-18  Aircraft,  the  planned  user  platform  for 
launching  the  weapon,  was  not  made  until  after  the  JSOW  program  progressed 
to  Phase  II,  EMD.  The  Integrated  Program  Summary  approved  for  JSOW  on 
April  1,  1992,  by  the  Assistant  Secretary  of  the  Navy  (Research,  Development 
and  Acquisition)  stated  that  the  requirements  allocation  would  be  made  early  in 
EMD.  Because  of  the  uncertainty  concerning  the  allocation  of  software 
requirements,  it  was  not  clear  during  Demonstration  and  Validation  what 
software  capabilities  would  have  to  be  developed  by  the  JSOW  program  rather 
than  by  the  F/A-18  program. 
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Prototype  software  was  not  developed  and  tested  during  Phase  I,  Demonstration 
and  Validation,  for  the  shore  segment  of  the  FDS.  FDS,  which  will  be  used  to 
detect  movements  of  enemy  submarines,  is  composed  of  two  segments;  the 
underwater  segment  and  the  shore  segment.  The  system  software  will  reside  in 
the  shore  segment,  known  as  the  Shore  Signal  Information  Processing  Segment. 
Shore  segment  software  was  not  available  to  support  testing  of  underwater 
system  segment  hardware  because  of  the  29-month  difference  in  the 
development  schedule  for  the  two  segments.  The  underwater  segment  was 
approved  to  enter  Phase  II,  HMD,  in  September  1989  while  the  shore  segment 
did  not  enter  EMD  until  February  1992.  During  Demonstration  and  Validation 
for  the  underwater  segment,  previously  existing  shore  processing  equipment  and 
software  from  the  Sound  Surveillance  System  (the  predecessor  to  FDS)  was 
used  for  early  evaluation  of  underwater  segment  system  concepts.  At  the  time 
of  the  Demonstration  and  Validation  for  the  underwater  segment,  an  estimated 
50  percent  (approximately  800,000  lines  of  code)  of  the  software  for  the  shore 
segment  would  have  to  be  developed  while  the  other  50  percent  could  be 
procured  "off-the-shelf."  A  lower  risk  approach  to  the  development  of  the  FDS 
would  have  involved  developing  prototype  shore  segment  software  in 
conjunction  with  the  development  of  the  undersea  hardware  since  the 
two  segments  must  work  together  to  allow  the  FDS  to  meet  mission  needs.  If 
delays  or  failures  occur  in  the  EMD  of  the  shore  segment,  operational  capability 
of  the  overall  system  will  be  delayed. 

Planning  Software  Test  and  Evaluation.  The  imposition  of 
documentation  requirements  that  were  not  appropriate  for  Demonstration  and 
Validation  hampered  the  effectiveness  of  software  development  and  testing  on 
the  Navy  JSOW.  Insufficient  documentation  of  software  test  planning  also 
hampered  planning  effectiveness  on  the  JSOW,  Army  BAT,  and  Air  Force  F-22 
programs. 

The  effectiveness  of  the  software  prototyping  done  under  the  Demonstration  and 
Validation  contracts  for  JSOW  was  reduced  because  the  contracts  did  not 
provide  for  tailored  application  of  DoD  Standard  2167A,  "Defense  System 
Software  Development."  The  DoD  Standard  requires  that  its  provisions  be 
tailored  for  each  application  so  that  only  cost-effective  provisions  are 
implemented.  The  Demonstration  and  Validation  contracts  for  JSOW  included 
blanket  application  of  all  provisions  of  the  DoD  Standard.  The  Stand^d 
includes  provisions  for  extensive  documentation  of  software  packages,  which 
are  helpful  to  the  Government  when  programs  are  in  EMD  or  production  but  are 
counterproductive  when  prototyping  software.  This  application  of  the  Standard 
should  be  tailored  because,  during  Demonstration  and  Validation,  the  primary 
focus  of  software  effort  should  be  on  determining  whether  critical  algorithms  or 
sections  of  the  software  can  be  successfully  programmed.  Requiring  formal 
software  documentation  to  an  extent  appropriate  for  the  later  phases  of  the 
acquisition  takes  time  and  money  from  the  primary  objective  of  determining 
whether  necessary  capabilities  can  be  programmed. 

Planning  for  the  test  and  evaluation  of  system  software  for  the  BAT,  JSOW, 
and  F-22  had  been  insufficiently  documented  as  the  programs  transitioned  from 
Demonstration  and  Validation  into  EMD,  All  three  programs  had  completed 
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Demonstration  and  Validation  before  July  1992.  We  compared  the  Test  and 
Evaluation  Master  Plans  (TEMPs)  used  during  Demonstration  and  Validation 
for  the  programs  with  those  to  be  followed  during  EMD  and  found  that  the 
software  test  and  evaluation  plans  were  no  longer  a  separate  annex  in  the  later 
versions  of  the  TEMP.  Instead,  only  abbreviated  summaries  of  software  test 
planning  were  provided.  By  allowing  the  removal  of  the  annexes  proyiding 
software  test  and  evaluation  plans  from  the  TEMP,  we  believe  the  Military 
Departments  have  deemphasized  planning  for  the  development  and  testing  of 
software  when  software  development  and  integration  are  becoming  increasingly 
important  to  system  success.  While  planning  for  software  test  ^d  evaluation  is 
in  the  Computer  Resources  Life-Cycle  Management  Plan,  this  document  has 
neither  the  visibility  or  the  organizational  coordination  involved  with  the 
TEMP,  which  DoD  Instruction  5000.2  requires  as  part  of  the  documentation  for 
each  acquisition  milestone  review.  The  Computer  Resources  Life-Cycle 
Management  Plan  is  not  part  of  the  milestone  documentation  requirements 
established  by  the  Instruction.  Program  office  personnel  for  the  BAT  stated  that 
the  annex  for  software  test  and  evaluation  was  dropped  to  meet  DoD  Manual 
5000. 2-M  requirements  to  keep  the  TEMP  under  30  pages.  The  30-page  limit 
does  not  apply  to  annexes,  however,  since  the  Instruction  states  they  may  be 
used  as  ne^ed. 

Continued  inclusion  of  a  software  test  and  evaluation  plan  as  an  annex  to  the 
TEMP  is  important  to  the  successful  planning  and  execution  of  Demonstration 
and  Validation  as  well  as  for  successfully  building  on  Demonstration  and 
Validation  accomplishments  during  EMD.  The  Institute  for  Defense  Analysis 
Document  D-1097,  "Prototyping  Defense  Systems,"  prepared  for  USD(A&T)  in 
December  1991,  states  that  software  has  historic^ly  caused  difficulty  and  that 
software  integration  has  become  more  complicated.  Additionally,  the  document 
states  that  software  is  becoming  a  major  part  of  overall  system  costs,  adding  to 
technical  risk  and  increasing  the  importance  of  prototyping. 

Manufacturing.  We  determined  that  two  of  the  six  programs  reviewed, 
the  Army  BAT  and  the  Navy  FDS,  did  not  prototype  critical  manufacturing 
processes  during  Phase  I,  Demonstration  and  Validation.  The  r^uirement  to 
include  manufacturing  as  part  of  the  Demonstration  and  Validation  effort  has 
been  required  since  1984  under  DoD  regulations  relating  to  acquisition.  DoD 
Directive  4245.6,  "Defense  Production  Management,"  January  19,  1984, 
(cancelled  in  1991  by  DoD  Instruction  5000.2)  required  that  manufacturing 
voids  and  deficiencies  be  addressed  during  Demonstration  and  Validation 
through  manufacturing  technology  projects.  Those  projects  were  to  involve  the 
development  or  improvement  of  manufacturing  processes,  techniques,  and 
equipment  to  help  bridge  the  gap  between  system  feasibility  and  production. 
DoD  Instruction  5000.2  retained  the  requirement  for  including  manufacturing 
processes  as  one  focus  for  prototyping  efforts. 

Representatives  from  the  Army  BAT  Program  Office  stated  that  manufacturing 
processes  were  not  prototyped  during  Demonstration  and  Validation  because 
they  believed  that  critical  manufacturing  processes  refer  to  producibility  risk  and 
do  not  relate  to  technical  risk.  Additionally,  they  did  not  believe  they  could 
prototype  manufacturing  processes  until  system  design  was  firm.  The  BAT 
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program  entered  Phase  II,  EMD,  in  May  1991  and  the  system  contractor  is 
currently  studying  producibility  to  identify  what  manufacturing  processes  will 
be  critical.  The  most  critical  processes  we  identified  were  those  relating  to  the 
acoustic  sensor.  Based  on  guidance  in  DoD  4245. 7-M,  "Transition  from 
Development  to  Production,"  September  1985,  we  believe  critieal 
manufacturing  processes  should  have  been  identified  and  prototyped  with  the 
hardware  during  Demonstration  and  Validation.  The  DoD  4245. 7-M  states  that 
the  manufacturing  process  required  to  produce  an  item  significantly  influences 
the  design  approach  and  product  configuration.  The  guid^ce  calls  for  early 
involvement  of  production  and  manufacturing  personnel  while  system  design  is 
still  fluid. 

Prototyping  of  manufacturing  processes  for  the  underwater  segment  of  the  Navy 
FDS  Program  was  limited  because  the  Demonstration  and  Validation  effort 
focused  on  proving  out  the  system  concept  rather  than  the  system  technology. 
Under  the  acquisition  strategy  for  the  system,  hardware  from  the  predecessor  to 
FDS,  the  Sound  Surveillance  System,  was  used  during  Demonstration  and 
Validation  to  verify  the  feasibility  of  a  new  pattern  for  distributing  listening 
devices.  Development  of  hardware  for  the  FDS  was  largely  planned  for  Phase 
II,  EMD.  As  a  result,  there  was  no  opportunity  to  prototype  manufacturing 
processes  related  to  FDS  hardware  during  Demonstration  and  Validation. 
Although  the  prime  contractor  for  the  underwater  segment  of  the  program  had 
experience  with  commercial  and  milita^  underwater  systems,  FDS  required 
new  levels  of  miniaturization  and  durability  of  design.  Because  technics  Md 
manufacturing  risks  were  not  adequately  assessed  and  reduced  during 
Demonstration  and  Validation,  the  program  was  projecting  a  significant  cost 
overrun  on  contract  N00039-90-C-0077  with  American  Telephone  and 
Telegraph  Technologies  Incorporated  for  the  EMD  of  the  underwater  segment. 
As  of  December  8,  1993,  the  FDS  Program  Manager  was  projecting  cost 
overruns  between  $91  million  and  $105  million  on  this  cost-plus-incentive-fee 
contract  that  was  originally  negotiated  for  $389  million.  The  negative  cost 
variance  primarily  involves  fiber  optic  cable  production  delays  and  technical 
issues. 

Competitive  Prototyping.  The  program  documentation  concerning  provisions 
for  competitive  prototyping  showed  that  one  of  the  six  programs  reviewed,  the 
Navy  JSOW,  had  not  met  provisions  for  waiver  of  the  requirement  to  perform 
competitive  prototyping  as  required  by  United  States  Code  and  DoD  Instruction 
5000.2.  Additionally,  we  determined  that  another  program,  the  OSD 
UTTMDS,  could  have  increased  the  potential  for  benefits  accruing  from 
competitive  prototyping  at  the  subcontractor  level. 

Waiver  of  Competitive  Prototyping  Requirement.  The  Navy  JSOW 
program  did  not  follow  the  provisions  of  title  10,  United  States  Code,  s^tion 
2365,  which  requires  use  of  a  competitive  prototype  acquisition  strategy  in  the 
development  of  a  major  weapon  system  (or  a  subsystem  of  such  system)  unless 
the  Secretary  of  Defense  submits  a  written  notification  explaining  why  the  use 
of  competitive  prototyping  is  not  practicable. 
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Acquisition  Plan  AIR  88-21,  approved  for  the  JSOW  on  July  1,  1988,  by  the 
Commander,  Naval  Air  Systems  Command,  describes  an  acquisition  approach 
that  includes  competition  but  clearly  precludes  the  use  of  prototypes.  The 
Acquisition  Plan  describes  a  competitive  Demonstration  and  Validation  that  was 
to  culminate  in  the  delivery  of  a  "paper  concept  and  a  level  1  data  package." 
This  plan  was  followed  by  the  Milestone  I,  Concept  Demonstration/Validation, 
decision  (now  Concept  Demonstration  Approval)  in  June  1989  and  the  issuance 
of  contracts  to  three  competing  contractors.  The  statement  of  work  for  contract 
N000i9-89-C-0076,  issued  to  Texas  Instruments,  Incorporated,  on  June  29, 
1989,  stated  that  "This  statement  of  work  stops  short  of  requiring  Prototype  or 
Advanced  Development  Model  design,  fabrication,  and  test."  The  statement  of 
work  provides  that  risk  reduction  efforts  were  limited  due  to  limitations  in 
funding.  The  same  statement  of  work  was  used  for  contracts  with  the  other 
two  competing  contractors.  Although  it  was  evident  in  the  Acquisition  Plan 
released  almost  a  year  before  Milestone  I  approval  that  the  Navy  did  not  plan  to 
use  competitive  prototyping  for  JSOW,  notification  and  justification  through  a 
waiver  request  was  not  forwarded  through  the  Secretary  of  Defense  to  Congress 
as  required  by  title  10,  United  States  Code,  section  2365.  Section  2365  expired 
September  30,  1991,  more  than  2  years  after  the  Milestone  I  approval  for 
JSOW.  In  October  1992,  the  competitive  prototyping  requirement  was 
reinstated  in  title  10,  United  States  Code,  section  2438,  "Major  Programs: 
Competitive  Prototyping."  In  section  2438,  the  USD(A&T)  was  given  authority 
to  approve  competitive  prototyping  waiver  requests  submitted  by  the  Military 
Departments. 

Competitive  Prototyping  by  Subcontractors.  Our  review  of  program 
acquisition  strategies  showed  that  two  programs  reviewed,  the  Army  BAT  and 
the  OSD  UTTMDS  programs,  planned  to  limit  competitive  prototyping  to  the 
subcontractor  level.  While  restricting  competitive  prototyping  to  the 
subcontractor  level  focuses  the  competitive  effort  on  key  subsystems,  thus 
reducing  program  cost,  the  winning  subcontractor  must  be  selected  based  on  the 
best  interest  of  the  Government.  While  the  Army  BAT  program  was 
establishing  the  necessary  provisions  to  control  the  selection  of  the  winning 
subcontractors,  the  OSD  UTTMDS  Program  had  not. 

The  acquisition  strategy  for  the  UTTMDS,  as  presented  in  the  Integrated 
Program  Summary  dated  November  12,  1991,  states  that  the  cost  of  carrying 
more  than  one  contractor  through  Demonstration  and  Validation  would  be 
prohibitive.  To  reduce  program  risk,  on  November  23,  1992,  the  UTTMDS 
Program  Manager  insert^  provisions  into  Change  2  to  the  Scope  of  Work  SW- 
K-50-90  to  contract  DASG  60-92-C-0101  for  the  Theater  High  Altitude  Area 
Defense  portion  of  the  system  to  require  the  contractor  to  include  dual  sourcing 
of  critical  components  as  part  of  the  contractor's  risk  management  plan. 
Change  2  to  the  contract’s  scope  of  work  was  then  followed  on  May  18,  1993, 
by  Change  5,  which  stated  that  the  contractor  should  select  sources  for 
components  based  on  which  source  had  the  most  cost-effective  and  technically 
acceptable  product.  While  money  was  saved  by  limiting  the  competitive 
prototyping  effort,  the  choice  of  the  winning  subcontractor  and  the  relative 
weights  given  to  cost  versus  technical  considerations  in  making  the  choice  will 
be  outside  the  control  of  the  Government. 
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The  Army  BAT  program  provides  a  positive  example  of  how  the  Government 
can  maintain  control  over  the  source-selection  process  while  using  competitive 
prototyping  among  subcontractors  under  a  single  prime  contract.  The  BAT 
Program  Office  planned  to  use  competitive  prototyping  at  the  subcontract  level 
as  part  of  the  Phase  I,  Demonstration  and  Validation,  of  a  pre-planned  product 
improvement  effort.  The  goals  of  the  pre-planned  product  improvement  are  to 
increase  the  lethality,  countermeasure  resistance,  and  sensor  capability  and 
selective  target  attack  features  of  the  basic  BAT  system. 

Four  contractors  will  competitively  build  prototypes  of  the  terminal  seeker  for 
the  BAT  Pre-planned  Product  Improvement  to  determine  which  subcontractor 
will  be  the  source  for  the  seeker.  Competition  will  be  reduced  to 
two  subcontractors  after  a  design  review  9  months  into  Demonstration  and 
Validation.  Final  selection  will  be  made  after  the  36-month  effort  involving 
drop  testing  of  prototype  seekers.  To  maintain  control  over  subcontractor 
competition,  the  BAT  Program  Office  has  provisions  in  the  Request  for 
Proposals  for  the  Pre-planned  Product  Improvement,  which  the  BAT  Program 
Manager  stated  will  allow  the  Government  to  establish  the  selection  criteria  for 
choosing  the  subcontractors.  The  provisions  also  provide  that  no  authorization 
can  be  given  to  proceed  with  a  subcontract  until  the  Government  has  approved 
the  subcontractor  selection.  We  believe  that  the  actions  planned  for  the  BAT 
Pre-planned  Product  Improvement  effort  meet  the  intent  of  DoD  4245. 7-M, 
"Transition  From  Development  to  Production,”  September  1985,  which  states 
that  the  effective  management  of  subcontractors  needs  more  emphasis  within 
industry  and  in  the  Government's  management  of  prime  contractors  if  there  is  to 
be  a  smooth  transition  to  production. 

Early  Operational  Assessments  of  Prototyping  Efforts.  Program 
Early  Operation^  Assessments  (EGAs)  showed  that  assessments  for  three  of  the 
six  programs  evaluated,  the  Army  BAT  and  the  Navy  JSOW  and  FDS,  were  not 
adequately  supported  by  test  and  evaluation  of  prototypes.  The  EGAs  ^e 
evaluations  of  operational  effectiveness  and  suitability  made  during 
Demonstration  and  Validation  by  the  independent  test  and  evaluation 
organizations  of  the  Military  Departments.  DoD  Instruction  5000.2  defines 
operational  effectiveness  as  the  overall  degree  of  mission  accomplishment  of  a 
system.  Gperational  suitability  is  defined  as  the  degree  to  which  a  system  can 
be  satisfactorily  placed  in  field  use  with  consideration  of  availability, 
compatibility,  reliability,  maintainability,  and  manpower  and  logistic 
supportability.  DoD  Instruction  5000.2  requires  that  prototypes  confirm  the 
feasibility  of  a  specific  design  approach  to  satisfy  the  mission  need  and  to 
achieve  minimum  acceptable  operational  performance  requirements  before 
progressing  to  Phase  II,  EMD.  The  EGA  is  the  means  to  determine  whether 
acceptable  performance  is  achievable  based  on  the  recommended  design 
approach. 

The  EGA  performed  by  the  Navy's  Gperational  Test  and  Evaluation  Force 
(OPTEVFGR)  for  the  JSGW  was  primarily  based  on  the  results  of  modeling  and 
simulation.  While  modeling  and  simulation  is  valuable  in  determining  how  a 
weapon  will  work  against  various  threat  defenses  and  in  estimating  lethality,  its 
use  assumes  that  th^e  system  will  work  at  some  given  level  of  efficiency. 
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Specifically,  the  EOA  of  the  Baseline  Advanced  Interdiction  Weapon  System 
(now  JSOW))  October  3,  1991,  listed  the  following  limitations  that  precluded 
resolution  of  critical  operational  issues  and  affected  the  ability  to  draw 
conclusions  regarding  the  operational  effectiveness  and  suitability  of  the  system: 

o  No  hardware  was  available  for  actual  flight  testing.  Testing 
consisted  chiefly  of  a  computer  modeling  and  simulation  assessment. 

o  One  model  used  in  testing  was  produced  by  one  of  the 
three  competing  contractors  and  may  not  have  adequately  represented  the 
performance  characteristics  of  actual  system  hardware. 

o  Terrain  data  were  not  used  in  scenario  modeling. 

o  Anti-aircraft  artillery  was  not  used  in  threat  simulation;  only 
surface-to-air  missiles  were  simulated  as  attacking  the  system. 

Additionally,  the  simulation  and  assessment  exercises  done  by  OPTEVFOR 
were  primarily  limited  to  operational  effectiveness  issues  and  did  not  include  the 
system  suitability  issues.  During  the  EOA,  the  assessment  of  JSOW's 
compatibility  with  current  and  proposed  mission-planning  systems  and  with  the 
A-6  and  F/A-18  aircraft  were  qualitative.  Mission-planning  systems  and 
planned  aircraft  platforms  were  not  compatible  with  JSOW  at  the  time  of  the 
EOA,  but  operator  experience  indicated  they  could  be  if  properly  modified. 
OPTEVFOR  did  not  assess  the  results  of  the  contractors'  testing  because  the 
EOA  was  performed  when  contractor  testing  was  ongoing.  The  Developmental 
Test  and  Evaluation  Report  summarizing  the  results  of  contractor  testing  was 
not  prepared  until  December  1991,  2  months  after  issuance  of  the  EOA.  Also, 
OPTEVFOR  personnel  stated  that  the  independent  test  organization  had  not 
historically  been  involved  with  the  assessment  of  aviation-related  programs  such 
as  JSOW  until  Phase  II,  EMD. 

The  EOA  performed  for  FDS  did  not  include  an  adequate  assessment  of  system 
technology  (hardware  and  software).  The  EOA,  like  the  overall  acquisition 
strategy  used  for  the  program  during  Demonstration  and  Validation,  focused 
instead  on  the  assessment  of  the  system  concept.  The  FDS  test  bed  was  first 
laid  out  in  October  1985,  and  the  OPTEVFOR  completed  the  EOA  in  July 
1989.  The  early  operational  testing  done  during  Demonstration  and  Validation 
was  designed  to  demonstrate  the  concept  for  a  new  pattern  for  distributing 
listening  devices  on  the  ocean  floor.  Hardware  from  the  existing  Sound 
Surveillance  System  was  used  to  support  the  testing  and  the  EOA.  The 
following  hardware  items  critical  to  the  success  of  the  FDS  were  not  prototyped 
and  assessed  during  Demonstration  and  Validation:  FDS  hydrophones  (listening 
devices),  multiplexer,  and  fiber  optic  cable.  The  effectiveness  of  the  EOA  was 
further  limited  since  operational  suitability  was  not  assessed  and  a  summary 
report  of  developmental  test  and  evaluation  was  not  available  for  OPTEVFOR 
personnel.  In  addition  to  limiting  the  scope  of  operational  testing,  the  FDS 
acquisition  strategy  did  not  allow  for  adequate  assessment  and  reduction  of 
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technical  risk  during  Demonstration  and  Validation.  As  a  result,  the  underwater 
segment  of  the  program  during  Phase  II,  EMD,  experienced  a  significant  cost 
overrun  as  discussed  under  "Manufacturing." 

Program  Baseline  Parameters.  We  found  that  the  Acquisition  Program 
Baselines  for  the  OSD  UTTMDS  and  Army  BAT,  two  of  the  three  prograins 
reviewed  that  were  still  involved  in  Phase  I,  Demonstration  and  Validation,  did 
not  establish  baseline  schedule  milestones  necessary  for  measuring  program 
progress  related  to  the  use  of  prototypes. 

The  baseline  established  January  28,  1992,  for  UTTMDS  did  not  include  the 
following  schedule  milestone  relating  to  use  of  prototypes: 

o  date  for  completion  of  technical  testing  and 

o  start  and  complete  dates  for  the  Early  Operational  Assessment. 

The  baseline  established  for  the  BAT  program  on  May  13,  1993,  provided 
schedule  milestones  for  the  basic  BAT  program,  which  is  in  Phase  II,  EMD, 
and  also  for  the  Pre-planned  Product  Improvement  effort,  which  is  in  Phase  I, 
Demonstration  and  Validation.  The  shortfalls  in  the  baseline  information 
relating  to  the  Demonstration  and  Validation  effort  included  neither  a  date  for 
completion  of  prototype  development  nor  start  or  complete  dates  for  technical 
testing  or  for  EOA. 

The  Pre-planned  Product  Improvement  Effort  is  scheduled  to  reach  Milestone 
II,  Development  Approval,  in  March  1997. 

Program  progress  against  Acquisition  Program  Baselines  is  reported  quarterly  in 
the  Defense  Acquisition  Executive  Summary  Report.  DoD  Instruction  5000.2 
and  DoD  Manual  5000. 2-M  require  this  progress  report  to  give  the  Defense 
Acquisition  Executive  early  warning  of  problems  in  meeting  interim  schedule 
milestones  in  the  Acquisition  Program  Baselines.  If  problems  are  reported,  the 
Defense  Acquisition  Executive  can  direct  corrective  action  before  the  program 
reaches  a  major  milestone  review  such  as  Milestone  II,  Development  Approval. 
When  schedule  milestones  are  not  established  for  delivery  of  all  or  key  parts  of 
prototype  hardware  or  for  start  and  finish  of  both  technical  testing  and 
operational  assessment,  the  baseline  reporting  process  cannot  completely  fulfill 
its  purpose  as  an  early  warning  system. 


Issues  With  the  Prototyping  Process 


The  primary  causes  of  the  deficiencies  identified  in  the  prototyping  efforts  for 
the  systems  reviewed  included: 
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o  lack  of  a  properly  structured  acquisition  planning  and  decision¬ 
making  process  for  balancing  cost,  schedule,  and  technical  risks  at  the 
Milestone  I  decision  point  to  derive  the  prototyping  strategy  for  Demonstration 
and  Validation; 

o  need  for  greater  involvement  of  ARPA  in  formulation  of  prototyping 
strategies;  and 

o  noncompliance  with  regulations  and  guidance  for  planning  and 
conducting  test  and  evaluation,  tracking  the  status  of  program  implementation, 
and  tailoring  implementation  of  regulations  to  meet  individual  program  needs. 

Acquisition  Planning  and  Decisionmaking  Process.  DoD  Instruction  5000.2 
and  DoD  5000-2-M,  "Defense  Acquisition  Management  Documentation  and 
Reports,"  do  not  adequately  define  a  planning  and  decisionmaking  process  for 
determining  how  prototyping  will  be  used  for  individual  MDAPs.  The  need  for 
improved  planning  and  decisionmaking  relating  to  prototyping  was  emphasized 
in  the  "Defense  Management  Report  to  the  President,"  June  12,  1989,  and  the 
RAND  Report  R-4161-ACQ,  "The  Nature  and  Role  of  Prototyping  in  Weapon 
Systems  Development,"  1992. 

The  Defense  Management  Report  stated  that  "Decisions  made  during  the  early 
phases  of  systems  development  —  including  those  that  involve  funds  and 
schedules  for  prototyping  and  testing  --  often  have  dramatic  consequences  for 
operational  performance  and  life-cycle  cost."  This  report  charged  the  Under 
Secretary  of  Defense  for  Acquisition  (now  USD[A&'n)  with  developing  and 
ensuring  rigorous  application  of  policies  that  support  sound  decisions  on  major 
programs  t&ough  Full-Scale  Engineering  Development  (now  Phase  II,  EMD). 

The  1992  RAND  Report,  which  was  prepared  under  sponsorship  of 
USD(A&T),  concluded  that  "there  is  no  single  approach  to  prototyping; 
effective  practice  involves  considerable  flexibility,  both  in  tailoring  a  particular 
strategy  to  the  needs  of  a  development  effort  and  in  using  the  resulting 
information."  The  report  recommended  that  prototyping  be  explicitly 
considered  as  part  of  the  strategy  for  development  of  weapon  systems,  but  that 
the  acquisition  policy  should  provide  broad  guidelines  on  prototyping  rather 
than  specifying  detailed  prototyping  strategies.  The  RAND  Rei)ort  advocated 
including  the  full  range  of  prototyping  considerations  in  a  rational  decision 
process. 

The  DoD  Instruction  5000.2  and  the  DoD  Manual  5000.2-M  do  not  meet 
intentions  of  the  Defense  Management  Report  and  RAND  recommendations 
because  the  Instruction  and  the  Manual  do  not  adequately  define  a 
decisionmaking  process  for  tailoring  prototyping  strategies  for  meeting  the 
unique  risk  reduction  needs  of  individual  programs.  Both  the  February  1991 
and  February  1993  updates  of  the  DoD  Instruction  5000.2  state  that  the 
prototyping  requirements  established  at  Milestone  I,  Concept  Demonstration 
Approval,  will  be  based  on  an  assessment  of  the  technical,  manufacturing,  and 
cost  risks  associated  with  the  proposed  concept  and  the  result  of  technology 
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demonstrations.  The  Instruction,  however,  neither  defines  prototyping  nor 
provides  a  process  for  determining  and  justifying  the  level  and  scope  of 
prototyping  deemed  to  be  appropriate  to  a  particular  program. 

Defining  Prototyping.  Although  DoD  Instruction  5000.2  states  that 
prototyping  "will  be  a  major  element  of  Phase  I  Demonstration  and  yalidation," 
the  Instruction  does  not  define  what  constitutes  a  prototype.  We  believe  that  an 
agre^-on  definition  is  necessary  for  effective  policy  implementation.  The 
RAND  Report  defined  a  prototype  as  a: 

tangible  product  (hardware  and/or  software)  that  allows  hands-on 
testing  in  a  realistic  environment  ....  In  scope  and  scale,  it 
represents  a  concept,  subsystem,  or  production  article  with  potential 
for  utility  ....  It  is  not  necessarily  a  complete  system,  but  rather 
focuses  on  those  high  risk  areas  critical  to  system  success. 

This  definition  should  be  expanded  to  include  use  of  electronic  simulation  as 
well  as  hardware  and  software.  In  this  context,  electronic  simulation  would 
involve  using  electronic  work  stations  to  demonstrate  the  performance  of 
potential  system  or  subsystem  designs. 

The  RAND  Report  also  discussed  the  use  of  prototyping  at  various  levels;  full 
system,  partial  system,  and  subsystem  levels.  Full  system  includes  all  key 
subsystems,  partial  system  would  include  only  one  or  two  subsystems  integrated 
into  a  platform  (such  as  an  engine  airframe  combination),  and  subsystem  would 
involve  subsystems  prototyped  independent  of  a  platform.  Scope  of  prototyping 
refers  to  whether  the  use  of  prototyping  involves  hardware,  software, 
manufacturing  processes,  or  some  combination  of  the  three.  Appendix  H  shows 
the  level  and  scope  of  prototyping  in  systems  we  evaluated. 

Determining  Appropriate  Prototyping  Strategy.  The  DoD  Instruction 
5000.2  does  not  provide  a  decisionmaking  process  for  determining  and 
justifying  the  level  and  scope  of  prototyping  planned  for  a  particular  program. 
The  Instruction  only  requires  that  information  on  planned  use  of  competitive 
prototyping  be  presented  in  the  Acquisition  Strategy  Report,  which  is  an  annex 
to  the  Integrated  Program  Summary  at  Milestone  I.  If  use  of  competitive 
prototyping  is  not  planned,  the  Instruction  requires  preparation  of  a  waiver 
presenting  the  reasons  it  is  impractical  to  comply  with  this  requirement.  Use  of 
competition  is  only  one  decision  that  must  be  made  with  regard  to  prototypes. 
The  choices  concerning  the  level  and  scope  of  prototyping  in  a  program  are  of 
greater  importance. 

Impediments  to  the  Use  of  Prototyping.  Major  impediments  to 
expanded  use  of  prototyping  acquisition  strategies  during  Phase  I, 
Demonstration  and  Validation,  are  the  preliminary  costs,  funding  requirements, 
and  schedule  impact.  Those  preliminary  costs  must  be  weighed  relative  to  the 
potential  benefit  of  ensuring  that  the  best  alternatives  are  selected  for  entry  into 
Phase  II,  EMD.  Premature  entry  into  EMD  before  essential  prerequisites  in 
design,  testing,  and  production  processes  are  demonstrated  significantly 
increases  program  risk.  Tight  funding  limitations  during  Demonstration  and 
Validation  resulted  in  key  events  that  should  support  the  Milestone  II  decision 
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being  deferred  until  after  the  Milestone  II  decision  is  made.  In  essence, 
underfunding  Demonstration  and  Validation  merely  transfers  the  cost  and  risk  to 
EMD  where  problems  receive  considerably  more  attention  due  to  the  level  of 
DoD's  financial  commitment.  Prototyping  acquisition  strategies  during  the 
Demonstration  and  Validation  phase  will  likely  impact  program  schedules.  We 
could  not  quantify  this  impact  because  we  could  not  assess  the  schedule  impact 
of  the  absence  of  prototyping  on  the  EMD  phase.  Performing  prototyping 
before  the  Milestone  II  decision  could  reduce  the  effort  required  during  EMD; 
however,  too  many  variables  are  in  program  management  to  evaluate  this 
impact.  The  RAND  Corporation  concluded  prototyping  appeared  to  increase 
the  time  from  Milestone  I,  Concept  Demonstration  Approval,  to  first 
operational  delivery  by  about  12  months. 

Existing  Guidance.  Limitations  in  the  DoD  Instruction  5000.2 
guidance  relating  to  planning  the  use  of  prototyping  were  identified  based  on 
our  review  of  the  Instruction,  conclusions  reached  in  the  RAND  Report 
regarding  prototyping  policy,  and  the  deficiencies  in  the  use  of  prototyping 
observed  in  five  of  the  six  systems  evaluated.  The  RAND  Report  concluded 
that  while  the  conceptual  framework  provided  by  DoD  Instruction  5000.2  for 
decisionmaking  on  competitive  prototyping  was  adequate,  simil^  guidance 
should  be  applied  to  decisionmaking  on  level  and  scope  of  prototyping  as  well. 
The  Report  stated  that  some  form  of  prototyping  is  almost  always  appropriate 
and  that  the  "burden  of  proof"  should  be  on  those  who  argue  that  a  prototyping 
activity  is  unnecessary  or  impractical.  In  cases  where  full-scale  articles  are  too 
expensive  or  technically  impractical,  subsystem  prototyping  may  still  be 
appropriate.  The  RAND  Report  also  concluded  that  policy  should  provide  for 
including  the  full  range  of  prototyping  considerations  in  the  decisionmaking 
process.  When  prototyping  was  not  deemed  necessary,  the  decisionmaking 
process  suggested  would  allow  for  explicit  rationalization  as  to  why  prototyping 
was  deem^  inappropriate. 

The  RAND  Report  was  based  on  the  February  1991  versions  of  the  DoD 
Instruction  5000.2  and  DoD  Manual  5000. 2-M.  Changes  made  in  the  February 
1993  update  of  the  Instruction  did  not  include  the  RAND  recommendations  for 
expanding  the  policy  guidance  to  provide  a  decisionmaking  framework  and 
documentation  requirements  to  support  planning  the  level  and  scope  of  all 
prototyping  efforts  similar  to  that  required  for  use  of  competition.  In  addition. 
Part  4  of  the  DoD  Manual  5000.2-M  still  contains  confusing  language  regarding 
the  justification  for  not  using  competitive  prototyping.  Change  1  to  DoD 
Manual  5000.2-M  promulgated  under  a  DoD  Publication  System  Change 
Transmittal,  issued  March  5,  1993,  states  that  the  following  language  was 
deleted  from  the  manual:  "Prepare  a  request  for  a  competitive  prototype 
strategy  waiver  for  milestone  decision  authority  approval,  under  authority 
delegated  by  the  Secretary  of  Defense,  specifying  the  basis  for  the  waiver  (see 
Part  12  for  competitive  prototype  strategy  waiver)."  The  last  portion  of  the 
above  language,  the  reference  to  Part  12,  was  never  deleted  so  the  Manual's 
discussion  on  justification  for  not  using  competitive  prototyping  now  refers  the 
reader  to  Part  12  which,  after  Change  1,  states  "Reserved  for  Future  Use." 
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Revising  Prototyping  Guidance.  Prototyping  can  be  made  a  more 
effective  risk-reduction  tool  by  revising  DoD  Instruction  5000.2  and  DoD 
Manual  5000.2-M  to: 

o  Outline  a  decisionmaking  process  for  tailoring  a  prototyping 
strategy  to  meet  the  risk-reduction  needs  of  specific  programs. 

o  Require  a  prototyping  plan  to  be  prepared  before  Milestone  I 
that  will  specify  the  level  and  scope  of  planned  prototyping  as  well  as  provide 
explicit  explanation  of  those  areas  where  prototyping  was  deemed  inappropriate. 

Additional  changes  to  guidance  and  regulation  to  support  the  recommended 
revisions  to  DoD  Instruction  5000.2  and  DoD  Manual  5000.2-M  would  include 
the  development  of  a  handbook  to  provide  guidance  for  preparation  of 
prototyping  plans. 

Decisionmaking  Process.  The  Instruction  and  the  Manual 
should  be  modified  to  provide  a  conceptual  framework  for  decisionm^ng ' 
concerning  planned  prototyping  strategies.  While  we  recognize  that  no  single 
prototyping  strategy  fits  all  programs,  basic  guidelines  are  needed  to  determine 
when  prototyping  is  appropriate  and  what  types  of  prototypes  would  be  effective 
and  efficient  in  reducing  risk  in  a  particular  program.  At  a  minimum,  policy 
should  provide: 


o  general  definitions  of  system  level,  partial  system 
level,  and  subsystem  level  prototyping  and 

o  considerations  for  determining  the  appropriate  level 
of  prototyping  (system,  partial  system,  or  subsystem)  for  hardware,  software, 
and  manufacturing  processes. 

The  RAND  Report  advocated  a  costs  versus  benefits  approach  to  making 
prototyping  decisions.  RAND  stated  that  the  benefits  of  prototyping  must  be 
weighed  against  the  costs  of  prototyping.  According  to  RAND,  those  benefits 
include  r^uced  technical  risk;  identification  of  critical  system  integration 
issues;  increased  accuracy  of  cost,  schedule,  and  performance  estimates;  and 
allowing  necessary  design  changes  to  be  made  early.  RAND  stated  that  costs  of 
prototyping  include  increased  preliminary  cost  and  a  slightly  longer  time  needed 
(about  12  months  average)  to  reach  first  operational  delivery.  We  could  not 
confirm  the  schedule  impact  of  prototyping  because  of  the  numerous  variables 
that  affect  program  outcome  and  the  many  differences  between  individual 
programs.  This  type  of  cost  versus  benefits  analysis  can  be  applied  at  the 
system,  partial  system,  and  subsystem  levels  for  hardware,  software,  and 
manufacturing  processes. 

The  Institute  for  Defense  Analysis  (IDA)  in  IDA  Document  D-1097, 
"Prototyping  Defense  Systems,"  December  1991,  provided  additional  guidance 
on  when  and  how  to  prototype.  This  document  was  prepared  for  the  Under 
Secretary  of  Defense  for  Acquisition  (now  USD[A&T]).  The  IDA  Document 
states  that  prototyping  should  be  done  for  systems  (or  subsystems)  that  involve: 
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o  new  performance  and  manufacturing  technology  for 

the  contractor(s), 

o  high  cost  per  unit  and  large  quantity,  and 
o  long  lead  time  or  high  cost  to  correct  potential 

unforeseen  problems. 

Prototyping  Plan.  To  provide  milestone  decision  authorities  the 
rationale  behind  the  proposed  prototyping  strategy  for  each  MDAP,  DoD 
Instruction  5000.2  should  require  that  the  program  manager  prepare  a 
prototyping  plan  as  part  of  the  documentation  for  the  Milestone  I  decision.  The 
prototyping  plan  should  be  based  on  the  knowledge  gained  during  Phase  0, 
Concept  Exploration,  and  the  best  judgment  of  experienced  engineers  and 
managers  applied  within  the  framework  of  the  amended  policy  guidance  as 
discussed.  The  prototyping  plan  should  document  how  prototyping  will  be  used 
to  reduce  the  technical,  cost,  and  schedule  risks  of  transitioning  from 
Demonstration  and  Validation  to  EMD.  The  plan  should  specifically  require 
justification  for  not  pursuing  a  prototyping  acquisition  strategy.  Documentation 
should  describe: 


o  the  extent  of  planned  development  of  system,  partial 
system,  or  subsystem  prototypes  of  hardware,  software,  and  manufacturing 
processes; 


o  use  of  competitive  prototyping  at  the  prime  and 

subcontractor  levels; 


o  developmental  test  and  evaluation  of  prototypes  to  be 
completed  before  EMD; 

o  the  operational  effectiveness  and  suitability 
demonstrations  to  be  completed  using  prototypes  to  support  the  Early 
Operational  Assessment; 

0  system  engineering  events  defined  in  Military 
Standards  499-A  and  152 1-B  that  will  be  based  on  prototypes  and  their  design 
documentation  (for  example,  the  extent  of  planned  design  reviews  on 
prototypes); 


o  how  modeling,  simulation  (sometimes  known  as 
electronic  prototyping),  and  producibility  analyses  will  supplement  and  enhance 
the  use  of  prototypes; 


o  how  requirements  of  DoD  Standard  2167A,  "Defense 
System  Software  Development,"  February  1988,  will  be  tailored  to  provide  for 
software  prototyping;  and 

o  the  trade-off  analysis  to  support  decisions  not  to  use 
prototyping  or  to  limit  its  use  to  the  partial  system  or  subsystem  levels. 
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Contractor  Input  to  Prototyping  Plans.  We  found  input  from 
potential  Demonstration  and  Validation  contractors  to  be  essential  for  planning 
the  prototyping  activities  to  be  in  the  formal  acquisition  strategy  approved  by 
the  milestone  decision  authority  at  Milestone  I.  The  first  version  of  the  plan 
should  be  prepared  before  issuance  of  the  Request  for  Proposals  for  the 
Demonstration  and  Validation  contract.  Program  managers  should  ensure  that 
solicitations  for  Demonstration  and  Validation  contracts  request  that  contractor 
proposals  identify  where  prototyping  will  be  used.  Revisions  to  the  prototyping 
plan  should  be  based  on  contractor  input  provided  in  response  to  the  Request  for 
Proposals.  To  accomplish  these  objectives,  the  DoD  Instruction  5000.2  should 
be  revised  to  include  language  on  prototyping  stating  that  solicitation  proposals 
should  include  a  complete  description  of  the  prototyping  plan  required  to  be 
submitted  with  contractor  proposals.  Additionally,  a  Data  Item  Description 
should  be  used  to  describe  the  contents  of  the  contractor  prototyping  plan. 
Program  managers  can  use  these  methods  to  solicit  contractor  prototyping 
approaches  and  assess  the  most  beneficial  uses  of  prototyping  as  part  of  the 
evaluation  process  leading  to  contract  award. 

Prototyping  Handbook.  A  handbook  should  be  developed  to 
provide  guidance  to  program  managers  for  preparation  of  prototyping  plans  for 
programs  approaching  Milestone  I,  Concept  Demonstration  Approval.  The 
handbook  guidance  should  address  cost  versus  benefits  consideration  of  system, 
partial  system,  or  subsystem  prototyping  for  hardwwe,  software,  and 
manufacturing  processes,  as  well  as  competitive  prototyping,  and  the  use  of 
modeling  and  simulation  to  enhance  prototyping  effort.  The  handbook  would 
act  as  a  supplement  to  the  revised  guidance  discussed  for  DoD  Instruction 
5000.2  and  DoD  Manual  5000.2-M. 

ARPA  Involvement  in  Formulating  Prototyping  Strategies.  Another  factor 
contributing  to  the  deficiencies  observed  in  prototyping  strategies  is  that  ARPA 
has  not  assumed  an  expanded  role  in  prototyping  as  intended  by  the  Packard 
Commission.  ARPA,  under  the  authority,  direction,  and  control  of  the  Director 
for  Defense  Research  and  Engineering,  serves  as  the  central  research  and 
development  organization  of  the  DoD  with  a  primary  responsibility  to  maintain 
U.S.  technological  superiority  over  potential  adversaries.  The  Packard 
Commission  Report  recommended  that  the  role  of  Defense  Advanced  Research 
Projects  Agency  (now  ARPA)  be  expanded  to  include  the  mission  of  stimulating 
greater  emphasis  on  prototyping  Defense  systems.  ARPA  was  to  accomplish 
this  expanded  role  by  actually  conducting  prototype  projects  that  embody 
technology  that  might  be  incorporated  in  joint  or  selected  programs  and,  on 
request,  assist  the  Military  Departments  in  their  prototyping  programs. 
Dr.  David  Packard,  Chairman  of  the  Commission,  clarified  those 
recommendations  in  a  letter  to  the  Secretary  of  Defense  in  April  1989.  The 
letter  was  also  signed  by  Dr.  William  J.  Perry,  then  a  member  of  the  Ad  Hoc 
Advisory  Group  on  Defense  Advanced  Research  Project  Agency’s  Role  in 
Prototyping  and  now  Secretary  of  Defense.  The  importance  the  Packard 
Commission  attached  to  prototyping  as  part  of  the  acquisition  process  is 
documented  by  three  quotations  from  the  April  1989  letter: 
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We  believe  you  [Secretary  of  Defense]  have  an  unparalleled 
opportunity  to  make  a  major  improvement  in  the  defense  acquisition 
system  by  supporting  prototyping  throughout  the  department. 


The  use  of  prototyping  is  fundamental  to  the  success  of  this  process, 
for  only  by  building  and  testing  prototypes  can  real  performance  be 
demonstrated  and  actual  costs  be  established. 


[PJrototyping  can  save  substantial  time  because  only  general 
performance  specifications  are  required,  the  benefits  of  competition 
can  be  used  to  minimize  costs  and  encourage  creativity,  and  a  sound 
base  for  the  next  Phase  can  be  established  by  testing  real  hardware. 


The  letter  discussed  the  use  of  prototyping  during  two  Phases:  Phase  I, 
Demonstration  of  System  Feasibility  and  Military  Worth  (equivalent  to  current 
Phase  I,  Demonstration  and  Validation),  and  Phase  11,  Full-Scale  Development 
Through  Low-Rate  Initial  Production  (equivalent  to  current  Phase  11,  EMD). 
The  letter  stated  that  ARPA  should  play  a  substantial  role  in  Phase  I 
prototyping,  particularly  in  extension  of  its  own  system  developments. 

ARPA  began  initiatives  in  the  late  1980s  to  increase  ARPA  involvement  in 
prototyping  in  response  to  the  Packard  Commission  recommendations. 
However,  the  Military  Departments  contested  ARPA  involvement  and  the 
Agency  never  received  the  appropriate  funding  to  allow  it  to  become  involved  in 
prototyping.  The  only  notable  exceptions  were  the  ARPA  assistance  in  the 
prototyping  efforts  for  the  Army  Tactical  Missile  System  and  the  Air  Force 
Joint  Surveillance  and  Target  Attack  Radar  System.  Current  ARPA  work  is 
focused  primarily  on  technology-based  projects  and  advanced  technology 
demonstrations. 

Those  efforts  are  precursors  to  the  prototyping  efforts  that  occur  during 
Demonstration  and  Validation  and  in  which  the  Packard  Commission 
recommended  ARPA  involvement. 

ARPA's  present  work  in  advanced  technologies  provides  the  Agency  with  the 
ability  to  provide  independent,  technology-driven  input  to  the  planning  of 
prototyping  strategies  for  major  systems.  To  meet  the  intention  of  the  Packard 
Commission  that  ARPA  "should  have  the  additional  mission  of  stimulating  a 
greater  emphasis  on  prototyping  Defense  systems,"  DoD  Instruction  5000.2 
should  require  ARPA  to  review  and  report  on  the  Acquisition  Strategy  Reports, 
with  emphasis  on  prototyping  plans,  for  all  Acquisition  Categories  I  and  11 
programs  before  Milestone  I,  Concept  Demonstration  Approval.  The 
Instruction  should  also  require  ARPA  to  review  and  report  to  the  Milestone 
Decision  Authority  concerning  the  need  for  prototyping  as  a  risk-reduction 
method  for  major  modifications  or  upgrades  of  Acquisition  Categories  I  and  II 
programs  approaching  Milestone  IV,  Major  Modification  Approv^. 
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Compliance  With  Regulation  and  Guidance.  We  believe  that  the  third  major 
contributing  factor  to  deficiencies  observed  in  prototyping  strategies  was 
compliance  with  existing  regulations  and  guidance.  Military  Departments  have 
not  fully  complied  with  existing  regulations  and  guidance  for  planning  and 
conducting  test  and  evaluation,  tracking  the  status  of  program  implementation, 
and  tailoring  implementation  of  regulations  to  meet  individual  program  needs. 

Planning  and  Conducting  Test  and  Evaluation.  The  deficiencies 
relating  to  the  test  and  evaluation  of  prototypes  occurred  because  operational 
testing  was  not  planned  and  executed  to  fully  support  early  operational 
assessments  and  comply  with  the  requirements  established  in  DoD  Instruction 
5000.2.  Demonstration  of  operational  capabilities  of  weapon  systems  during 
Demonstration  and  Validation  has  long  been  a  part  of  the  acquisition  process  as 
defined  by  DoD  Instruction  5000.2.  The  September  1987  version  required  that 
operational  effectiveness  and  suitability  goals  and  thresholds  be  "met  and 
reviewed"  before  Milestone  II,  Full-Scale  Development  decision  (now  c^led 
Development  Approval).  This  requirement  supported  the  Defense  Acquisition 
Board's  Milestone  II  deliberations  concerning  system  cost  and  affordability 
versus  system  effectiveness  meeting  operational  requirements.  The  Febru^ 
1993  update  of  the  Instruction  requires  that  prototyping  include  the  opportunity 
for  an  assessment  of  operational  effectiveness  and  suitability  by  the  operational 
test  organization. 

The  July  1989  Defense  Management  Report  to  the  President  also  eniphasized 
the  importance  of  using  prototypes  to  evaluate  operational  capabilities  of 
systems.  The  Report  stated  that  schedules  and  plans  should  provide  for  early 
test  and  evaluation  of  prototype  hardware  to  prove  concept,  performance,  and 
suitability  in  a  realistic  operational  environment. 

Maximizing  the  effectiveness  of  operational  test  and  evaluation  and  increasing 
the  meaningfulness  of  early  operational  assessments  require  close  and 
continuous  interface  between  the  program  office  developing  a  system  and  the 
Military  Department's  operational  test  organizations.  The  F-22  Aircraft 
Program  provides  a  positive  example  of  effective  interface  between  the  program 
manager  and  the  operational  test  organization.  The  program's  April  1991 
TEMP  documented  that  the  Air  Force  Operational  Test  and  Evaluation  Center 
conducted  a  continuing  operational  assessment  of  the  program  from  November 
1986  until  the  June  1991  EOA  was  issued  to  support  Milestone  II,  Development 
Approval.  During  this  5-year  assessment  process,  the  operational  test 
organization  evaluated  progress  in  many  phases  of  the  Demonstration  and 
Validation  activities.  Specifically,  the  assessment  included: 

o  review  of  contractor's  avionics  prototype  capabilities  and 

o  participation  in  contractor's  flight  test  program,  contractor 
and  program  office  worting  groups,  mission  simulation  efforts,  and  relevant 
maintenance  demonstrations. 
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The  June  1991  EGA,  which  supported  the  Milestone  II,  Development  Approval, 
was  not  a  one-time  review  but  the  product  of  continued  effective  interface 
between  the  operational  test  organization  and  the  program  office. 

Tracking  the  Status  of  Program  Implementation.  OSD  UTTMDS 
and  Army  BAT,  the  two  programs  having  acquisition  program  baselines  that  did 
not  include  program  schedule  events  necessary  for  measuring  program  progress 
relating  to  prototyping,  did  not  comply  with  the  DoD  Manual  5000. 2-M 
requirement  for  establishing  baseline  schedule  information  adequate  to  describe 
the  program.  DoD  Manual  5000.2-M  provides  the  formats  for  preparing 
acquisition  program  baselines,  which  include  perform^ce,  schedule,  and  cost 
parameters.  For  the  schedule  parameters  to  be  established  at  Milestone  I,  the 
Manual  requires  that  baseline  entries  be  established  for  "date  of  Demonstration 
and  Validation  contract  award  [and]  early  operational  assessment  (start  - 
complete)."  The  Manual  further  states  that  "the  minimum  dates  [are]  required 
in  each  baseline  but  are  rarely  sufficient  to  describe  the  program."  To  fill  the 
void  in  the  program  schedule  descriptions,  the  Manual  suggests  dates  for 
"prototype  development  complete  [and]  technical  test  (start  -  complete)"  also  be 
established  in  the  baseline. 

Because  prototyping  is  a  major  element  of  Phase  I,  Demonstration  and 
Validation,  the  DoD  Manual  5000.2-M  should  be  modified  to  require  those 
dates  as  part  of  the  Milestone  I  Concept  Baselines  whenever  prototyping  is 
planned.  This  change  to  DoD  Manual  5000.2-M  will  ensure  that  acquisition 
managers  have  an  adequate  reference  point  for  measuring  and  reporting  the 
status  of  program  implementation  and  that  baseline  schedule  information 
includes  dates  necessary  to  describe  the  program  adequately,  as  required  by 
DoD  Manual  5000.2-M. 

Tailoring  Regulations.  Not  consistently  applying  this  requirement 
contributed  to  the  shortfalls  noted  in  software  prototyping.  DoD  Standard 
2167A,  Defense  System  Software  Development,  requires  tailoring  the  Standard 
to  meet  specific  program  requirements.  Most  contractors  we  survey^  believed 
that  prototyping  software  during  Phase  I,  Demonstration  and  Validation,  should 
be  focused  on  critical  software  algorithms.  The  contractors  believed  that 
imposing  the  full  software  documentation  and  reporting  requirements  described 
in  the  Standard  can  be  counterproductive  during  Demonstration  and  Validation. 


Impact  of  Prototyping 


Increased  program  risk  is  the  primary  impact  of  the  deficiencies  noted  in  the 
prototyping  process.  When  full  execution  of  a  prototyping  strategy  that  has 
been  matched  to  the  particular  needs  of  a  program  does  not  occur  in  Phase  I, 
Demonstration  and  Validation,  then  Milestone  II,  Development  Approval,  may 
occur  when  it  has  not  been  sufficiently  demonstrated  that  critical  technologies 
can  be  incorporated  into  the  design  or  proven  that  processes  critical  to  the 
system  concept  are  at  an  acceptable  level  of  risk.  Milestone  II,  Development 
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Approval,  represents  a  commitment  to  translate  the  design  approach  developed 
during  Demonstration  and  Validation  into  a  stable,  producible,  and  cost- 
effective  design  that  can  satisfy  mission  need  and  meet  operational  performance 
requirements.  When  misconceptions  concerning  the  expected  capabilities  of  a 
system  concept  and  the  ability  to  understand  and  incorporate  the  critical 
technologies  exist  at  Milestone  if,  then  program  risk  increases  the  likelihood  of: 

o  large  additional  expenditures  during  Phase  II,  HMD,  to  overcome 
unforeseen  technical  problems; 

o  cancellation  of  systems  for  engineering  and  development  due  to  poor 
performance; 

o  difficulties  in  transitioning  into  production;  and 


o  accepting  systems  that  do  not  fully  meet  mission  requirements. 

Execution  of  an  appropriate  prototyping  strategy  during  Demonstration  and 
Validation  improves  the  accuracy  of  cost  and  sch^ule  estimates  for  both  EMD 
and  production  of  a  system.  The  information  gained  from  prototyping 
organizations  allows  program  managers  to  make  more  educated  estimates  of  the 
costs  and  the  time  needed  to  engineer  and  produce  a  system.  In  the  same 
manner,  prototyping  provides  program  managers  the  technical  information  to 
more  accurately  predict  technical  performance  of  a  program. 

The  benefits  of  prototyping  were  documented  in  the  IDA  report,  prepared  for 
USD(A&T)  in  December  1991.  D-1097,  "Prototyping  Defense  Systems," 

stated  that  both  development  and  production  cost  growth  were  significantly 
lower  for  prototyped  programs  than  for  non-prototyped  programs.  IDA 
examined  52  major  acquisition  programs  over  32  years,  ending  December  31, 
1987.  The  examination  included  aircraft,  tactical  munitions,  satellites,  and 
helicopters  and  reported  the  following  differences  in  percentage  of  cost  growth 
between  the  17  programs  that  prototyped  and  the  35  that  did  not: 


Growth  Area 


Prototyped  Non-Prototyped 

(in  percent!  (in  percent) 


Development  Cost  Growth  17  62 

Production  Cost  Growth  29  55 


Percentage  cost  growth  was  derived  by  comparing  the  cost  estimates  for  each 
program  at  Milestone  II  with  the  latest  estimates  provided  in  the  December  1987 
Selected  Acquisition  Report.  Cost  growth  was  adjusted  for  changes  in 
production  quantity  and  inflation.  We  could  not  independently  confirm  the 
percentages  of  cost  growth  on  programs  that  used  and  did  not  use  prototyping. 
In  particular,  we  could  not  confirm  that  the  reported  differences  in  cost  growth 
had  a  direct  causal  relationship  to  prototyping  efforts.  Nevertheless, 
prototyping  efforts  should  provide  the  information  necessary  to  develop  more 
accurate  cost,  schedule,  and  performance  forecasts. 
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The  IDA  Report  also  discussed  possible  negative  effects  prototyping  can  have 
on  programs.  Those  effects  included: 

o  Extra  time  required.  According  to  the  Report,  prototyped  programs 
took  about  2  years  longer  to  progress  from  Milestone  I,  Concept  Demonstration 
Approval,  to  Initial  Operational  Capability  than  programs  that  did  not  use 
prototyping. 

o  Increases  preliminary  costs  of  programs. 

o  Delays  major  funding  commitment.  The  large  scale  commitment  of 
funding  at  the  Milestone  II,  Development  Approval,  is  viewed  as  necessary  to 
"lock-in"  support  for  the  program. 

Revising  the  DoD  Instruction  5000.2  and  DoD  Manual  5000.2-M  to  provide 
cost^enefit  consideration  for  determining  the  appropriate  level  of  prototyping 
(system,  partial  system,  or  subsystem)  for  hardware,  software  and 
manufacturing  processes  will  maintain  or  enhance  the  documented  benefits  of 
prototyping  while  minimizing  the  negative  impacts.  The  result  of  this 
systematic  approach  should  be  a  prototyping  strategy,  documented  in  an 
approved  prototyping  plan  that  focuses  prototyping  on  those  areas  where 
additional  technical  information  is  most  needed  to  reduce  program  risk  and 
effectively  integrates  prototyping  with  other  development  techniques  such  as 
modeling  and  simulation.  Increased  emphasis  on  choosing  the  most  appropriate 
prototyping  strategy  will  ensure  that  time  and  money  spent  on  prototyping  will 
be  limited  to  what  has  been  determined  as  necessary  to  reduce  technical  risk  to 
an  appropriate  level.  Additionally,  delaying  funding  commitment  allows  the 
Government  to  keep  its  options  open  as  suggested  in  the  IDA  report. 


Conclusions 


Effective  prototyping  to  reduce  program  risk  in  transitioning  from 
Demonstration  and  Validation  to  EMD  can  only  occur  through  the  development 
and  implementation  of  a  structured  decisionmaking  process  that  will  result  in  a 
documented  and  executable  prototyping  plan.  This  prototyping  plan  should 
incorporate  the  best  judgment  of: 

o  experienced  program  managers  and  engineers  within  Government 
and  industry, 

o  research  scientists  at  ARPA  who  are  familiar  with  critical 
technologies  associated  with  the  planned  weapon  system,  and 

o  test  personnel  at  the  operational  test  organizations  of  the  Military 
Departments  who  must  ensure  that  planned  prototyping  will  provide  opportunity 
for  early  operational  assessment  of  operational  effectiveness  and  suitability. 
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The  designation  of  ARPA  for  a  leadership  role  in  prototyping  decisions  at  the 
Milestone  I  decision  point  is  consistent  with  the  Agency  mission  as  described 
under  "ARPA  Involvement  in  Formulating  Prototyping  Strategies."  Expanding 
the  ARPA  role  will  ensure  that  decisions  regarding  prototyping  are  based  on 
sound  technical  information.  ARPA  has  cognizance  for  advanced  technology 
demonstrations  that  should  be  transitioned  to  development  programs. 

We  recognize  that  no  single  approach  or  application  of  prototyping  can  be  used 
for  all  programs.  Therefore,  the  prototyping  plan  should  be  developed  to  fit  the 
unique  needs  of  a  particular  program.  To  support  effective  planning  of 
prototyping  efforts,  DoD  Instruction  5000.2  should  be  modified  to  provide  for 
determining,  through  systematic  costs  versus  benefits  analysis,  whether 
prototyping  should  be  used  at  the  system,  partial  system,  or  subsystem  level, 
and  for  specific  decisions  on  the  subsystems  to  be  prototyped.  The 
determination  process  should  be  applied  to  the  hardware,  software,  and  critical 
manufacturing  processes  involved  in  the  acquisition.  The  prototyping  plan 
should  also  describe  the  planned  use  of  modeling  and  simulation  to  supplement 
prototyping  effort.  A  handbook  containing  detailed  guidance  for  formulating 
the  prototyping  plan  should  be  developed  to  supplement  the  general  guidance  in 
the  DoD  Instruction  5000.2.  In  all  cases,  as  stated  in  the  RAND  study,  the 
burden  of  proof  should  be  on  those  who  believe  that  prototyping  is  unnecessary 
or  impractical. 

Once  the  prototyping  plan  is  developed,  we  believe  the  following  controls  and 
procedures  will  promote  effective  execution  of  Phase  I,  Demonstration  and 
Validation: 

o  Establish  specific,  quantifiable  exit  criteria  to  define  events  or 
achievements  using  prototypes  that  should  occur  before  the  program  transitions 
to  Phase  II,  EMD. 

o  Tailor  requirements  of  DoD  Standard  2167A,  Defense  System 
Software  Development,  or  applicable  commercial  standards  to  include  those 
provisions  that  are  cost-effective  to  the  development  of  prototype  software. 

0  Close  and  continuous  interface  among  the  program  offices  and  the 
operational  test  organizations  of  the  Military  Departments. 

o  Effective  tracking  of  the  status  of  program  implementation  through 
program  office  reporting  against  acquisition  program  baselines,  approved  by  the 
Defense  Acquisition  Executive,  which  provides  schedule  milestones  relating  to 
the  development  of  prototypes  and  the  starting  and  completion  of  technical  tests. 

Further,  the  use  of  prototyping  fully  supports  DoD  initiatives  to  expand  the  use 
of  performance  specifications  in  the  acquisition  of  weapon  systems. 
Performance  specifications  permit  contractors  to  more  thoroughly  examine 
alternatives  for  affordably  meeting  military  requirements  by  not  mandating  a 
particular  design.  The  specifications  are  particularly  important  during 
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Demonstration  and  Validation  and  entirely  consistent  with  a  prototyping 
acquisition  strategy  to  assess  the  potential  for  technological  approaches  to  meet 
performance  objectives. 

Our  recommendations,  if  fully  implemented,  will  significantly  elevate  the 
significance  of  the  Milestone  I,  Demonstration  Approval,  decision  jmd  provide 
for  a  more  gradual  transition  from  Demonstration  and  Validation  into  EMD. 
An  expanded  Demonstration  and  Validation  phase  that  includes  significant 
prototyping  will  permit  funding  to  be  committed  more  incrementally  and  based 
on  demonstrated  program  progress  instead  of  the  more  traditional  entry  into 
EMD  accompanied  by  a  single  major  increment  in  funding.  Further,  use  of 
prototypes  facilitates  the  preparation  of  more  accurate  cost  and  operational 
effectiveness  analyses  and  life-cycle  cost  estimates. 

Many  programs  have  encountered  significant  developmental  problems  during 
EMD.  Those  programs  had  Milestone  II  decisions  when  greater  developmental 
risk  was  justified  based  on  the  urgency  to  respond  to  potential  threats.  In 
general,  DoD  has  stated  that  higher  levels  of  risk  are  no  longer  justifiable. 
Under  those  circumstances,  expanded  use  of  prototyping  to  reduce  program  risk 
and  improve  the  quality  of  Milestone  II  decisions  is  both  logical  and  essential. 
Prototyping  enhances  the  quality  of  the  Milestone  II  decision  by  providing  the 
basis  for  more  accurate  forecasts  of  program  cost,  schedule,  and  performance. 

Although  we  recognize  the  need  to  avoid  unnecessary  regulatory  guidance,  the 
present  guidance  for  planning  and  implementing  prototyping  strategies  must  be 
expanded.  Prototyping,  along  with  related  activities  accomplished  during 
Phase  I,  Demonstration  and  Validation,  provides  the  knowledge  and  experience 
that  enhances  likelihood  of  program  success  in  all  subsequent  and  much  more 
expensive  acquisition  phases.  Therefore,  DoD  acquisition  management  must 
overcome  impediments  to  expanded  use  of  prototyping  by  placing  the  burden  of 
proof  on  program  managers  who  do  not  recommend  prototyping.  Implementing 
the  processes  and  procedures  discussed  will  facilitate  effective  and  inforrned 
decisionmaking  for  the  initial  and  critical  phases  of  the  system  acquisition 
process. 


Recommendations,  Management  Comments,  and  Audit 
Response 


1.  We  recommend  that  the  Under  Secretary  of  Defense  for  Acquisition 
and  Technology: 

a.  Revise  DoD  Instruction  5000.2  to: 

(1)  Provide  a  definition  of  prototyping  as  it  applies  to 
Phase  I,  Demonstration  and  Validation,  that  encompasses  prototyping  done 
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at  the  system,  partial  system,  and  subsystem  levels;  involves  hardware, 
software,  manufacturing  processes;  and  includes  electronic  simulation  and 
design  analysis. 

Director,  Acquisition  Program  Integration  (API),  Comments.  The  Director 
nonconcurred,  stating  that  prototyping  was  a  well-known  concept.  The  full  text 
of  management's  comments  is  in  Part  IV. 

Audit  Response.  We  disagree  with  management's  assertion  that  prototyping  is 
a  well-known  concept.  We  believe  a  definition  is  needed  now  more  than  ever. 
The  summary  section  of  RAND  Report  R-4161-ACQ,  "The  Nature  and  Role  of 
Prototyping  in  Weapon  System  Development,"  1992,  stated  that  confusion 
regarding  terms,  including  prototyping,  has  historically  led  to  inconsistencies  in 
acquisition  policy.  The  Report  further  stated  that  evidence  suggested  that  lack 
of  a  clear  definition  of  prototyping  has  contributed  to  a  lack  of  consensus  on 
prototyping  policy. 

A  compounding  factor  in  defining  prototyping  is  the  development  of  modeling 
and  simulation  methods  that  may  supplement  or  be  in  lieu  of  more  tradition^ 
hardware  prototypes.  During  our  review,  the  Director  of  Product  Integrity  in 
the  Office  of  the  Assistant  of  the  Navy  (Research,  Development  and 
Acquisition)  agreed  that  there  is  no  adequate  definition  of  prototyping.  He  also 
stated  that  any  prototyping  definition  should  include  the  use  of  electronic 
simulation  (or  electronic  prototyping)  as  well  as  the  more  traditional  use  of 
hardware.  We,  therefore,  ask  management  to  reconsider  its  position  in  response 
to  the  final  report. 

(2)  Require  program  managei'S  to  submit  a  prototyping  plan 
as  part  of  the  Acquisition  Strategy  Report  for  the  Milestone  I,  Concept 
Demonstration  Approval.  The  plan  must  be  based  on  cost,  schedule,  and 
technical  risks;  describe  the  intended  use  of  prototypes  in  risk-reduction 
efforts  at  the  system,  subsystem,  or  component  levels;  and  include 
consideration  of  prototyping  hardware,  software,  and  manufacturing 
processes  as  well  as  the  use  of  competitive  prototyping  at  contractor  and 
subcontractor  levels. 

Director,  API,  Comments.  The  Director  nonconcurred,  stating  that  this  plan 
is  already  an  inherent  part  of  the  Acquisition  Strategy  Report  and  the  TEMP. 
He  believed  that  additional  oversight  was  not  justified  by  the  findings  and  are 
counter  to  the  acquisition  streamlining  required  by  Executive  Order  12861.  He 
also  believed  that  adding  to  program  manager  workload  is  contrary  to  National 
Performance  Review  efforts.  Finally,  the  Director  stated  that  before  accepting 
far-reaching  recommendations,  the  findings  of  the  Acquisition  Task  Force  on 
Modeling  and  Simulation,  which  is  examining  the  feasibility  of  using  coniputer- 
generated  models  in  lieu  of  prototypes,  should  be  considered.  The  Director 
stated  that  the  Task  Force  effort  should  be  complete  within  90  days  (from 
June  3,  1994). 

Audit  Response.  We  recognize  that  the  Acquisition  Strategy  Report  and  the 
TEMP  contain  some  elements  that  could  be  integrated  into  a  prototyping  plan. 
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However,  neither  document  provides  complete  and  consolidated  documentation 
describing  the  extent  of  planned  development  and  use  of  system,  partial  system 
or  subsystem  prototypes  of  hardware,  software,  or  manufacturing  processes. 
Specifically  the  Acquisition  Strategy  Plan  and  the  TEMP  are  not  required  to 
document: 

i 

o  a  trade-off  analysis  made  to  support  decisions  not  to  use  prototyping 
or  to  limit  its  use  to  the  partial  system  or  subsystem  levels; 

o  how  requirements  of  DoD  Standard  2167A,  "Defense  System 
Software  Development,"  or  similar  commercial  standards  will  be  tailored  to 
provide  for  software  prototyping;  and 

o  system  engineering  events  defined  in  Military  Standards  499-A  and 
1521-B  that  will  be  based  on  prototypes  and  their  design  documentation. 

With  regard  to  management's  contention  that  our  recommendation  runs  counter 
to  Executive  Order  12861  and  the  National  Performance  Review,  we  do  not 
believe  those  initiatives  prohibit  or  discourage  thorough  planning.  Improving 
the  quality  of  information  provided  at  the  Milestone  I  decision  will  reduce  the 
need  for  additional  and  unscheduled  oversight  during  Demonstration  and 
Validation  and  better  position  programs  for  achieving  Milestone  II, 
Development,  approval.  We  agree  with  management  that  implementation  of 
our  recommendation  would  be  best  accomplished  after  the  Task  Force  on 
Modeling  and  Simulation  completes  its  efforts.  We,  therefore,  ask  that 
management  reconsider  its  position  in  response  to  the  final  report. 

(3)  Require  the  Advanced  Research  Projects  Agency  to 
report  to  the  milestone  decision  authority  on  its  review  of  Acquisition 
Strategy  Reports,  with  primary  emphasis  on  prototyping  plans,  for  all 
Acquisition  Categories  I  and  II  acquisition  programs  before  Milestone  I, 
Concept  Demonstration  Approval. 

Director,  API,  Comments.  The  Director  nonconcurred,  stating  that  the  report 
did  not  provide  strong  rationale,  other  than  the  1986  Packed  Commission 
Report,  for  the  recommendation.  He  further  stated  that  although  ARPA  is 
technology  based,  it  does  not  have  the  knowledge  or  expertise  that  is  resident  in 
the  Military  Departments  and  is  not  resourced  or  staff^  to  be  the  single  DoD 
focal  point  for  prototyping. 

Audit  Response.  We  believe  that  ARPA  is  capable  and  can  provide  critical 
input  into  the  formulation  of  prototyping  strategies  at  the  Milestone  I  decision, 
especially  in;  those  cases  where  ARPA  was  involved  in  pre-Milestone  I  research 
and  development  effort.  DoD  Directive  5105.41,  Defense  Advanced  Research 
Projects  Agency,  January  25,  1989,  states  that  ARPA  serves  as  the  central 
research  and  development  organization  of  the  Department  of  Defense  and  is  to 
stimulate  a  greater  emphasis  on  prototyping  in  Defense  systems  by  conducting 
prototype  projects  (designated  as  advanced  technology  demonstrations  in  DoD 
Instruction  5000.2)  that  embody  technology  that  might  be  incorporated  in 
Military  Department  programs. 


35 


Use  of  Prototyping  in  Acquisition  Strategies 


In  terms  of  knowledge  of  Military  Department  programs,  an  ARPA  official 
estimated  that  ARPA  has  historically  performed  some  level  of  technology 
demonstration  work  on  about  40  percent  of  the  acquisition  programs  going  to 
Milestone  1.  He  stated  that  the  degree  of  ARPA  involvement  has  ranged  from 
system-level  demonstrations  of  the  technologies  that  later  became  the  Army 
Tactical  Missile  System  and  the  Air  Force  Joint  Target  Attack  Radar  System  to 
demonstrations  of  technology  to  be  used  at  the  subsystem  or  component  levels. 
In  cases  where  ARPA  has  done  significant  pre-Milestone  I  work,  ARPA 
personnel  would  have  as  much  or  more  expertise  concerning  technologies 
critical  to  system  development  as  Military  Department  personnel  and  would  be 
able  to  provide  valuable  and  independent  input  to  the  milestone  decision 
authority  concerning  the  adequacy  of  planned  usage  of  prototypes  for  risk 
reduction  during  Phase  I,  Demonstration  and  Validation. 

Similarly,  we  believe  ARPA  could  provide  valuable  input  to  Milestone  I 
decisions  in  cases  where  ARPA  has  not  been  directly  involved  in  technical 
demonstrations.  In  these  instances  ARPA  could,  within  limitations  of  available 
staffing  and  with  priority  given  to  Acquisition  Category  I  programs,  serve  as  a 
focal  point  to  eliminate  duplicative  or  unnecessary  developmental  effort  during 
Phase  I,  Demonstration  and  Validation.  ARPA's  input  would  also  help  identify 
the  need  for  prototyping  and  for  developing  an  acceptable  prototyping  strategy. 
These  benefits  could  accrue  from  ARPA's  wide  knowledge  of  technical  efforts 
completed  or  ongoing  within  the  Military  Departments,  private  business  entities, 
educational  institutions,  and  foreign  nations. 

In  respect  to  the  Acquisition  Strategy  Report,  we  recommend  that  ARPA's 
review  be  focused  on  the  prototyping  plans  rather  than  the  entire  Acquisition 
Strategy  Report.  We,  therefore,  request  that  management  reconsider  its  position 
in  response  to  the  final  report. 

(4)  Require  the  Advanced  Research  Projects  Agency  to 
report  to  the  milestone  decision  authority  concerning  the  need  for 
prototyping  as  a  risk-reduction  method  for  major  modifications  or 
upgrades  of  Acquisition  Categories  I  and  II  programs  before  Milestone  IV, 
Major  Modification  Approval. 

Director,  API,  Comments.  The  Director  nonconcurred  with  this 
recommendation  for  the  same  reasons  given  in  response  to  Recommendation 
l.a.p).  Additionally,  he  questioned  whether  ARPA  is  the  correct  organization 
to  review  producibility,  a  very  important  issue  at  Milestone  IV. 

Audit  Response.  As  in  our  audit  response  for  Recommendation  l.a.(3),  we 
believe  that  ARPA  can  provide  valuable  input  to  the  milestone  decision 
authority  in  support  of  the  Milestone  IV  decision.  Producibility  is  a  major  issue 
at  Milestone  IV  and  incorporation  of  technological  advances,  as  part  of  system 
modifications  or  upgrades,  is  a  key  component  of  the  producibility 
determination.  At  Milestone  IV,  the  milestone  decision  authority  is  to 
determine  which  acquisition  phase  should  be  entered  based  on  level  of  risk,  the 
adequacy  of  risk  management  planning,  and  the  level  of  resources  to  be 
committed.  In  this  respect,  ARPA  has  the  ability  to  provide  the  milestone 
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decision  authority  with  independent  input  concerning  the  need  for  prototyping  as 
a  risk  reduction  method  for  major  modifications  or  upgrades  of  Acquisition 
Categories  I  and  II  programs.  We,  therefore,  request  that  management 
reconsider  its  position  in  response  to  the  final  report. 

(5)  Require  the  Directors  of  Military  Departments' 
operational  test  and  evaluation  organizations  to  report  specifically  at 
Milestone  I  on  system  and  subsystem-level  prototyping  requirements  needed 
for  optimal  operational  assessments  of  systems  suitability  and  effectiveness 
during  Demonstration  and  Validation. 

Director,  API,  Comments.  The  Director  partially  concurred,  stating  that  the 
Director,  Operational  Test  and  Evaluation,  sees  benefit  in  the  r^ommendation 
but  that  the  Army  and  the  Navy  question  how  this  plan  would  be  implemented. 

Audit  Response.  The  recommendation  can  be  implemented  by  requiring  the 
directors  of  the  Military  Department's  operational  test  and  evaluation 
organizations  to  submit  a  separate  assessment  of  prototyping  requirements  to  the 
Director,  Operational  Test  and  Evaluation.  This  assessement  would  address  the 
system  or  subsystem  level  prototyping  needed  for  optimal  operational 
assessments  of  systems'  suitability  and  effectiveness  during  Demonstration  and 
Validation.  We  request  that  management  consider  our  suggestion  for 
implementing  the  recommendation  in  responding  to  the  final  report. 

(6)  When  prototyping  is  used,  require  "Prototype 
Development  Complete"  and  "Technical  Test  (Start  -  Complete)"  to  be  part 
of  the  minimum  required  schedule  dates  for  Defense  Acquisition  Executive 
Concept  Baselines  for  programs  in  Phase  I,  Demonstration  and  Validation. 

Director,  API,  Conunents.  The  Director  concurred  with  the  recommendation 
but  did  not  provide  an  estimated  completion  date  for  revising  DoD  Instruction 
5000.2. 

Audit  Response.  In  response  to  the  final  report,  we  ask  that  management 
provide  an  estimated  completion  date  for  revision  of  the  Instruction. 

(7)  Require  specific,  quantifiable  exit  criteria  for  programs 
involving  the  use  of  prototyping  during  Phase  I,  Demonstration  and 
Validation. 

Director,  API,  Comments.  The  Director  nonconcurred,  stating  that  requiring 
exit  criteria  regardless  of  program  particulars  would  not  allow  the  program 
manager  and  the  Milestone  Decision  Authority  enough  leeway  to  run  programs 
in  a  manner  reflecting  good  business  practice.  He  further  stated  that  the 
recommendation  presupposes  that  all  programs  are  prototyped,  which  he 
believes  is  unlikely. 

Audit  Response.  DoD  Instruction  5000.2  states  that  exit  criteria  are  intended 
to  benefit  both  the  milestone  decision  authority  and  the  program  manager.  For 
the  milestone  decision  authority,  the  use  of  exit  criteria  offers  the  flexibility  to 
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set  execution  boundaries  for  each  phase  of  the  program  and  to  regulate  the 
amount  of  oversight  to  be  applied  during  the  phase.  For  the  program  manager, 
the  use  of  exit  criteria  offers  the  freedom  to  execute  key  events  during  a  phase 
without  the  formality  of  milestone  decision  authority  and  staff  reviews  except  at 
milestone  decisions. 

We  recognize  that  individual  exit  criteria  must  be  carefully  and  selectively 
applied  but  also  believe  that  establishing  some  type  of  specific,  quantifiable  exit 
criteria  for  programs  involving  the  use  of  prototypes  in  the  acquisition  strategy 
is  a  logical  supplement  to  the  general  decision  criteria  established  by  DoD 
Instruction  5000,2  for  the  Milestone  II,  Development  Approval,  decision. 
These  general  decision  criteria  include  the  requirement  that  prototyping  and 
demonstration  efforts  to  date  provide  reasonable  assurance  that  the  technologies 
and  processes  critical  to  success  are  attainable.  The  exit  criteria  would  support 
the  general  decision  criteria  by  requiring  specific,  measurable  accomplishments 
relating  to  performance,  technology,  or  events  involving  prototypes  that  can  be 
used  to  assess  program  readiness  to  move  to  EMD.  We  have  clarified  the 
recommendation  in  the  final  report  to  specify  that  exit  criteria  should  be 
required  for  those  programs  involving  the  use  of  prototyping  acquisition 
strategies  during  Demonstration  and  Validation.  We,  therefore,  request  that 
management  reconsider  its  position  in  response  to  the  final  report. 

(8)  Require  an  initial  design  review  to  be  completed  before 
Milestone  n,  Development  Approval,  based  in  part  on  the  results  of 
prototyping  for  programs  involving  prototyping  acquisition  strategies. 

Director,  API,  Comments.  The  Director  nonconcurred,  stating  that  the 
existing  decision  criteria  specified  in  DoD  Instruction  5000.2  for  Milestone  II 
make  the  recommended  change  unnecessary.  He  stated  that  the  current  version 
of  the  Instruction  requires  that  the  decision  authority  confirm  that  "Prototyping 
and  Demonstration  results  to  date  provide  reasonable  assurance  that  the 
technologies  and  processes  critical  to  success  are  attainable.”  Further,  the 
Director  stated  that  the  recommendation  presupposes  that  all  programs  are 
prototyped. 

Audit  Response.  Before  Milestone  II,  DoD  Instruction  5000.2  does  require 
that  the  milestone  decision  authority  confirm  that  prototyping  and  demonstration 
results  to  date  provide  reasonable  assurance  that  technologies  and  processes 
critical  to  success  are  attainable.  DoD  Instruction  5000.2  decision  criterion  also 
supports  the  need  to  implement  our  recommendation.  How  can  the  milestone 
decision  authority  confirm  the  general  criterion  of  "reasonable  assurance" 
without  some  specific  criteria  for  measuring  the  attainability  of  critical 
manufacturing  technologies  and  process?  Through  performance  of  an  initial 
design  review,  the  milestone  decision  authority  would  be  made  aware  of  known 
limitations  or  shortfalls  in  the  system  design,  along  with  contractor-propos^ 
solutions  for  overcoming  design  problems.  We  clarified  the  recommendation  in 
the  final  report  to  specify  its  applicability  to  only  those  programs  involving  a 
prototyping  acquisition  strategy.  We,  therefore,  ask  management  to  reconsider 
its  position  in  response  to  the  final  report. 
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(9)  Require  a  producibility  analysis  to  be  prepared  based  on 
prototypes  for  programs  involving  prototyping  acquisition  strategies. 

Director,  API,  Comments.  The  Director  nonconcurred,  stating  that  not  all 
programs  will  be  prototyped.  Further,  he  stated  that  a  producibility  analysis  is 
normally  prepared  during  EMD  to  support  Production  Readiness  Reviews  and 
the  analysis  is  based  on  the  EMD  model.  He  stated  that  prototypes  can  also  be 
used  for  input  into  the  analysis. 

Audit  Response.  DoD  Instruction  5000.2  r^uires  that  the  focus  of  prototyping 
be  on  assessing  and  reducing  the  risks  associated  with  integrating  available  and 
emerging  technologies  into  a  system  design  approach  to  satisfy  a  validated 
mission  need.  In  defining  this  requirement,  the  Instruction  states  that 
technologies  will  include  hardware,  software,  and  manufacturing  processes. 
For  Milestone  II,  the  decision  criteria  includes  the  requirement  that  prototyping 
and  demonstration  results  to  date  provide  reasonable  assurance  that  the 
technologies  (including  manufacturing)  and  processes  critical  to  success  are 
attainable.  The  producibility  analysis,  like  the  initial  design  review  discussed  in 
our  response  to  management  comments  on  Recommendation  l.a.(8),  would 
provide  the  milestone  decision  authority  at  Milestone  II  with  reasonable 
assurance  that  technologies  and  processes  critical  to  success  are  attmnable.  The 
product  of  this  analysis  would  identify  known  and  potential  production  problems 
along  with  contractor's  proposed  approaches  for  solving  the  problems.  We  have 
clarified  the  recommendation  in  the  final  report  to  specify  applicability  to  only 
those  programs  involving  a  prototyping  acquisition  strategy.  We,  therefore,  ask 
management  to  reconsider  its  position  in  response  to  the  final  report. 

(10)  Require  Requests  for  Proposals  for  Demonstration  and 
Validation  contracts  to  include  a  provision  for  proposing  the  level  of 
software  documentation  and  reporting,  under  DoD  Standard  2167A, 
"Defense  System  Software  Development,"  or  applicable  commercial 
standards,  that  contractors  deem  most  appropriate  through  applying  the 
principles  as  provided  in  DoD  Handbook  248,  "Guide  for  Application  and 
Tailoring  of  Requirements  for  Defense  Material  Acquisitions."  The 
proposal  documentation  from  each  contractor  should  provide  the  cost  and 
benefit  justification  for  the  level  for  the  type  of  documentation  proposed. 

Director,  API,  Comments.  The  Director  partially  concurred  with  draft 
Recommendation  l.a.(ll)  (renumbered  Recommendation  l.a.[10]).  He  stated 
that  it  is  inappropriate  to  require  the  use  of  a  DoD  Standard  when  commercial 
standards  are  preferable  in  accordance  with  Deputy  Secretary  of  Defense 
guidance.  ■ 

Audit  Response.  Management  comments  were  responsive  to  the  intent  of  the 
Recommendation  l.a.(lO).  Based  on  management's  comments,  we  revised  the 
recommendation  to  provide  a  wider  focus.  Our  revision  provides  for  proposing 
tailoring  to  the  DoD  Standard  or  commercial  standards  for  software 
documentation  and  reporting  as  contractors  deem  most  appropriate.  In  the 
proposal  documentation,  each  contractor  should  provide  the  cost  and 
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benefit  justification  for  the  tailored  documentation  proposed.  We,  therefore, 
request  management  to  provide  comments  on  our  revised  recommendation  in 
response  to  the  final  report. 

(11)  Revise  language  on  prototyping  to^  state  that  solicitation 
proposals  for  Phase  I,  Demonstration  and  Validation,  contracts  should 
include  a  complete  description  of  the  prototyping  plan  required  to  be 
submitted  with  the  contractor  proposals. 

We  added  Recommendation  l.a.(ll)  as  a  result  of  comments  provided  by  the 
Director,  Defense  Procurement,  and  the  Director,  Acquisition  Program 
Integration,  on  Recommendation  3.  in  the  draft  report.  We  request  the  Under 
Secretary  of  Defense  for  Acquisition  and  Technology  to  provide  comments  on 
Recommendation  l.a.(ll)  in  response  to  the  final  report. 

b.  Issue  a  handbook  providing  guidance  for  formulation  of 
prototyping  plans  for  programs  approaching  Milestone  I,  Concept 
Demonstration  Approval.  Handbook  guidance  should  cover  costs  and 
benefits  consideration  of  system,  subsystem,  or  component  prototyping  for 
hardware,  software,  and  manufacturing  processes;  competitive 
prototyping;  and  the  use  of  modeling  and  simulation  to  substitute  for  or 
refine  prototyping  effort. 

Director,  API,  Comments.  The  Director  nonconcurred,  stating  that  it  was  not 
clear  that  a  handbook  was  necessary  and  that  it  certainly  was  not  necessary  to 
revise  DoD  Instruction  5000.2  to  issue  a  handbook. 

Audit  Response.  We  do  not  agree  with  management's  assertion  that  a  clear 
need  does  not  exist  for  a  prototyping  handbook.  A  handbook  is  needed  because 
existing  DoD  guidance  does  not  provide  program  managers  enough  assistance  to 
determine  the  level  and  scope  of  prototyping  warranted  on  each  program.  The 
handbook  would  help  program  managers  to  effectively  perform  the  cost  versus 
benefits  consideration  of  system,  partial  system,  or  subsystem  prototyping 
options  for  hardware,  software,  and  manufacturing  processes.  Also,  the 
handbook  would  help  program  managers  understand  how  to  consider  the  use  of 
competitive  prototyping  and  determine  how  modeling  and  simulation  could  be 
integrated  into  the  prototyping  process.  To  be  useful,  the  handbook  should 
include  a  discussion  of  the  possible  benefit  and  cost  factors  that  can  enter  a 
prototyping  decision  and  discuss  the  possible  methodologies  to  weigh  these 
factors.  The  handbook  should  not  give  specific  weights  to  the  benefit  and  cost 
factors  or  to  mandate  how  each  factor  should  enter  the  decision  process.  In 
summary,  a  handbook  would  be  a  needed  supplement  to  existing  guidance  in 
DoD  Instruction  5000.2.  We  request  that  management  reconsider  its  position  in 
response  to  the  final  report. 

c.  Revise  DoD  Manual  5000.2-M  to: 

(1)  Delete  the  language  in  Part  4  that  cites  Part  12  as  an 
example  of  the  competitive  prototyping  strategy  waiver. 
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Director,  API,  Comments.  The  Director  nonconcurred  with  draft 
Recommendation  l.c.,  (renumbered  Recommendation  l.c.[l]),  stating  that 
Change  1  incorporated  the  requirement  to  include  the  rationale  for  not  using 
prototyping  strategy  as  part  of  the  Acquisition  Strategy  Report, 

Audit  Response.  Management  was  correct  in  asserting  that  Change  1  to  DoD 
Manual  5000.2-M  incorporated  the  requirement  to  include  the  rationale  in  the 
acquisition  strategy  report  when  it  is  not  practicable  to  use  a  competitive 
prototyping  strategy.  However,  Part  4  of  the  DoD  Manual  5000.2-M  still 
contains  confusing  language  regarding  the  justification  for  not  prototyping. 
DoD  Publication  System  Change  Transmittal,  issued  March  5,  1993,  that 
transmitted  Change  1,  states  that  the  following  language  was  deleted:  "Prep^e 
a  request  for  a  competitive  prototype  strategy  waiver  for  milestone  decision 
authority  approval,  under  authority  delegate  by  the  Secretary  of  Defense, 
specifying  the  basis  for  the  waiver  (see  Part  12  for  competitive  prototype 
strategy  waiver),"  The  last  portion  of  the  above  language,  the  reference  to  Part 
12,  was  never  actually  deleted  so  the  Manual's  discussion  on  justification  for 
not  using  competitive  prototyping  still  refers  the  reader  to  Part  12  that,  after 
Change  1,  states  "Reserved  for  Future  Use."  We  revised  the  Recommendation 
to  require  only  that  the  faulty  reference  be  removed  at  the  next  update  of  the 
manual.  We,  therefore,  request  management  to  provide  comments  on  the 
revised  recommendation  in  response  to  the  final  report. 

(2)  Require  a  software  test  and  evaluation  plan  to  be  included 
as  an  annex  to  the  Test  and  Evaluation  Master  Plan  in  support  of  all 
milestone  decisions  in  all  cases  where  software  is  determined  to  be  a 
moderate  or  high  risk  area  in  the  Annex  D,  Risk  Assessment,  of  the 
Integrated  Program  Summary. 

Director,  API,  Comments.  The  Director  nonconcurred  with  the  draft 
Recommendation  l.a.(10)  (renumbered  Recommendation  l.c. [2]).  He  stated 
that  software  test  and  evaluation  is  currently  a  required  integral  part  of  the 
TEMP. 

Audit  Response.  We  agree  with  management's  statement  that  the  current 
version  of  DoD  Instruction  5000.2  requires  software  test  and  e^valuation  to  be  a 
part  of  the  TEMP.  However,  we  still  believe  additional  guidance  is  needed 
concerning  how  the  software  test  and  evaluation  planning  should  be  documented 
in  the  TEMP.  As  a  result,  we  revised  the  recommendation  to  recommend  that 
DoD  Manual  5000.2-M  be  revised  to  provide  guidance  for  addressing  software 
test  and  evaluation  in  the  TEMP.  Precedence  validates  our  recommended  policy 
change.  Before  its  cancellation  in  1991,  DoD  Manual  5000.3-M-3,  "Software 
Test  and  Evaluation  Manual,"  November  1987,  provided  guidance  identifying 
specific  portions  of  the  TEMP  that  should  include  software  aspects  in  the 
planning  for  system  test  and  evaluation.  Specifically,  the  DoD  Manual 
5000.3-M-3  required  that: 

o  software  test  and  evaluation  planning  address  mission,  system,  and 
software  requirements  and  their  associated  support  requirements  and 
interrelationships; 
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o  the  TEMP  include  provisions  for  software  test  and  evaluation 
throughout  the  life-cycle  of  the  system; 

o  each  phase  of  evaluation  be  structured  to  provide  a  clear 
understanding  of  software  maturity,  to  form  the  basis  for  future  evaluations,  and 
to  establish  traceability  with  previous  evaluations;  and 

o  system-level  testing  be  designed  and  conducted  to  demonstrate  the 
extent  of  fault-free  or  fault-tolerant  software  performance  and  determine  the 
level  of  achieved  mission  requirements. 

In  addition,  the  DoD  Manual  5000.3-M-3  referenced  guidelines  on  how  to 
evaluate  TEMPs  to  ensure  that  software  was  adequately  addressed  and  gave 
examples  of  software  test  programs. 

Although  software  is  more  critical  to  system  success  than  ever,  DoD  Manual 
5000. 2-M  does  not  provide  the  type  of  guidance  described  above  for  integrating 
and  documenting  planned  software  test  and  evaluation  into  the  TEMP.  The 
cancelled  DoD  Manual  5000.3-M-3  discussed  two  options  for  integrating 
software  test  and  evaluation  into  the  TEMP:  embedding  the  software  test  and 
evaluation  details  into  the  body  of  the  TEMP  or  attaching  the  software  test  and 
evaluation  details  as  an  appendix,  while  not  repeating  items  addressed  in  the 
main  document.  Given  the  TEMP's  30-page  restriction  in  DoD  Manual 
5000.2-M  (excluding  appendixes),  the  documentation  of  planned  software  test 
and  evaluation  in  an  appendix  to  the  TEMP  appears  to  be  the  most  practical 
approach.  We  also  recognize  that  software  may  not  be  a  critical  area  on  every 
MDAP.  Therefore,  the  recommended  revision  to  DoD  Manual  5000.2-M 
should  be  limited  to  those  programs  where  software  was  rated  as  moderate  to 
high  risk  in  Annex'  D,  Risk  Assessment,  of  the  Integrated  Program  Summ^y. 
We,  therefore,  ask  that  management  provide  comments  on  our  revised 
recommendation  in  response  to  the  final  report. 

d.  Designate  an  office  of  primary  responsibility  for  determining 
whether  new  Data  Item  Descriptions  must  be  developed  to  allow  for 
submission  of  contractor  prototyping  plans  and  contractor  reports 
documenting  the  results  of  prototyping  risk  reduction  efforts. 

Audit  Response.  We  added  Recommendation  l.d.  in  response  to  Director, 
Defense  Procurement  and  Director,  Acquisition  Program  Integration,  comments 
received  on  Recommendation  3.  in  the  draft  report.  We,  therefore,  request  the 
Under  Secretary  of  Defense  for  Acquisition  and  Technology  to  provide 
comments  on  Recommendation  l.d.  in  response  to  the  final  report. 

Overall  Director,  API,  Comments  on  Recommendation  1.  The  Director 
agreed  with  our  overall  finding  that  prototyping  could  be  used  more  effectively 
to  reduce  risk  and  assess  how  design  approaches  will  operate  before  proceeding 
with  EMD.  However,  he  did  not  support  most  of  our  recommendations  because 
he  believed  the  recommendations  would  radically  reduce  the  flexibility  provided 
to  the  milestone  decision  authorities  under  DoD  Instruction  5000.2,  that  is,  to 
choose  an  acquisition  strategy  appropriate  to  the  specific  situation.  The  Director 
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further  stated  that  he  believed  that  our  recommendations  would  add  unnecessary 
oversight  and  appear  to  conflict  with  Secretary  of  Defense  guidance  concerning 
waiver  authority,  streamlining,  and  reduction  of  bureaucratic  regulations.  Also, 
he  stated  that  prototyping  requires  additional  up-front  money  and  that  our 
recommendations  could  hinder  the  Armed  Forces  by  diverting  funds  from 
production  to  mandatory  prototypes.  Finally,  he  believed  that  our  findings 
(specific  problems  cited  on  individual  MDAPs)  were  weak,  that  we  did  not 
provide  enough  information  on  the  Army  Comanche  and  Air  Force  F-22 
programs,  and  that  our  report  had  too  much  background  information  that  he 
already  was  aware  of. 

Audit  Response.  We  disagree  with  the  contention  that  our  recommendations 
would  radically  reduce  the  flexibility  provided  to  the  milestone  decision 
authorities  under  DoD  Instruction  5000.2  and  result  in  unnecessary  oversight  in 
conflict  with  Secretary  of  Defense  guidance  on  waiver  authority.  Rather  than 
reducing  flexibility  and  adding  oversight,  our  recommendations  are  designed  to 
provide  information  that  will  improve  the  quality  of  decisionmaWng  at 
Milestones  I,  II,  and  IV.  We  advocate  an  amended  policy  framework  that  not 
only  would  still  allow  program  managers  to  choose  the  acquisition  strategy  they 
believe  is  most  appropriate  but  also  requires  them  to  document  the  trade-off 
analysis  made  to  support  their  decisions.  If  carefully  planned  and  supported 
decisions  are  made  at  the  scheduled  milestones,  the  need  for  unscheduled 
oversight  between  the  milestones  will  be  reduced.  In  our  responses  to  the 
management  comments  on  individual  recommendations,  we  explained  the  need 
for  the  changes  to  DoD  Instruction  5000.2  and  the  benefits  of  increased  roles  in 
developing  prototyping  acquisition  strategies  by  the  Director,  ARP  A,  and  the 
Director,  Program  Analysis  and  Review. 

o  In  our  responses  to  management's  comments,  we  have  provided 
further  rationale  for  changing  policies  related  to  prototyping.  Additionally, 
Appendix  G.  documents  that  prototyping  was  too  limited  to  adequately  reduce 
risk  in  four  of  the  six  programs  we  reviewed.  The  four  programs  had 
significant  limitations  in  the  planning  or  execution  of  hardware  prototyping 
strategies.  Two  of  the  four  programs  also  had  limitations  in  software 
prototyping  strategies  and  two  programs  had  limitations  in  prototyping 
manuracturing  processes.  Also,  Early  Operational  Assessments  us^  to 
determine  whether  acceptable  performance  is  achievable  based  on  the 
recommended  design  approach  were  not  adequately  supported  by  the  test  and 
evaluation  of  prototypes  for  three  of  the  six  programs.  Additionally,  the  results 
of  our  review  reaffirmed  the  following  conclusions  of  earlier  studies  on 
prototyping  performed  by  RAND  and  the  IDA: 

o  Confusion  regarding  terms,  including  prototyping,  has 
historically  led  to  inconsistencies  in  acquisition  policy  and  strategies. 

o  Lack  of  a  clear  definition  of  prototyping  has  contributed  to  a 
lack  of  consensus  on  prototyping  policy. 
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o  Criteria  provided  by  DoD  Instruction  5000.2  for  determining 
whether  to  use  competitive  prototyping  could  be  applied  to  other  forms  of 
prototyping  activity  as  well. 

o  Important  and  challenging  programs  should  be  prototyped. 

o  We  recognize  that  prototyping  may  divert  additional  funds  to  system 
development;  however,  if  properly  planned  and  executed,  most  prototyping  will 
also  reduce  overall  program  cost.  Prior  research  by  the  IDA  documented  the 
positive  relationship  between  prototyping  and  reduced  program  cost  growth. 
Prototyping,  when  properly  executed,  also  provides  information  that  makes 
costs  more  predictable  and  thus  facilitates  more  accurate  budgeting  of  resources. 
Our  recommendations  are  intended  to  foster  informed  choices  concerning  the 
amount  of  prototyping  that  is  done  and  to  ensure  that  the  optimal  amount  of  risk 
and  cost  reduction  benefits  accrue. 

o  Management's  contention  that  our  draft  report  discussed  extensively 
acquisition  policies  and  prior  studies  with  which  the  Director,  Acquisition 
Program  Integration,  was  already  familiar  may  be  valid,  but  is  irrelevant.  A 
thorough  discussion  of  the  relevant  acquisition  policies  and  studies  is  necessary 
to  provide  all  readers  with  the  necessary  background  to  understand  the  problems 
relating  to  prototyping  and  the  need  for  implementing  our  recommendations. 
The  final  report  will  be  distributed  to  a  variety  of  users. 

o  We  disagree  with  management's  claim  that  our  draft  report  had  many 
inaccuracies  relating  to  the  UTTMDS,  JSOW,  and  FDS  programs  and  that  our 
report  was  unbalanced  because  it  largely  ignored  the  F-22  and  Comanche 
programs.  In  Appendix  I,  we  clarified  factual  information  on  UTTMDS, 
JSOW,  and  FDS.  In  the  draft  report,  we  cited  the  F-22  program  as  a  positive 
example  of  how  effective  interface  between  the  program  manager  and  the 
Military  Department's  operational  test  organization  can  support  successful 
progression  to  Milestone  II,  Development  Approval.  The  Comanche  Program 
was  in  Appendix  F  of  the  draft  report,  which  shows  that  prototyping  on  the 
program  was  too  limited  to  adequately  reduce  hardware  risk.  Although  we  did 
not  cite  the  Comanche  program  as  a  specific  example  in  Part  II  of  the  draft 
report,  we  determined  that  the  Comanche  Program,  like  the  BAT  Program,  was 
progressing  to  EMD  with  inadequate  hardware  risk  reduction.  Appendix  I 
provides  additional  information  related  to  the  limitations  of  prototyping  on  the 
Comanche.  For  brevity,  we  only  discussed  the  BAT  example  in  Part  II  of  the 
draft  report. 

The  complete  text  of  the  Director,  Acquisition  Program  Integration,  comments 
is  in  Part  IV.  The  audit  responses  to  the  Director's  comments  on  the  text  of  our 
finding  are  provided  in  Appendix  I. 

2.  We  reconunend  that  the  Director,  Program  Analysis  and  Evaluation, 
report  on  the  sufficiency  of  the  proposed  prototyping  plan  at  Milestone  I  in 
support  of  preparation  of  adequate  cost  and  operational  efficiency  analyses 
and  life-cycle  cost  estimates. 
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Director,  Program  Analysis  and  Evaluation,  Comments.  The  Director 
nonconcurred,  stating  that  the  report  contained  no  clearly  identifiable  statement 
of  findings  and,  to  the  extent  that  a  finding  can  be  inferred,  lacked  compelling 
supporting  rationale.  He  further  stated  that  it  seems  premature  to  change  DoD 
Instruction  5000.2  to  mandate  a  highly  structured,  prescriptive  prototyping  plan 
given  the  lack  of  compelling  rationale.  The  complete  text  of  management 
comments  is  in  Part  IV. 

Audit  Response.  We  have  difficulty  understanding  management's  comments 
that  the  report  did  not  clearly  identify  the  finding  condition  and  did  not  provide 
compelling  supporting  rationale.  The  finding  condition  is  clearly  stated  in  the 
first  sentence:  "Prototyping  could  be  used  more  effectively  in  the  acquisition 
strategies  for  major  Defense  acquisition  programs  to  identify  and  reduce  risk 
and  to  assess  whether  the  most  promising  design  approaches  will  operate  in  the 
intended  operating  environment  before  preceding  with  EMD."  The  finding  is 
supported  by  our  documentation  of  deficiencies  in  the  planning  and  execution  of 
the  prototyping  strategies  in  six  major  Defense  acquisition  programs  and  the 
results  of  the  numerous  earlier  studies  of  prototyping  presented  in  Part  II  of  the 
report  and  in  Appendix  D. 

o  Our  recommendations  would  result  in  a  structured  but  flexible 
methodology  for  determining  the  level  and  scope  of  prototyping  warranted  on 
each  program.  As  explained  in  our  report,  a  prototyping  plan  would  involve  the 
iterative  application  of  an  analytical  process  to  arrive  at  a  tailored  course  of 
action  suited  to  the  needs  of  a  given  program.  The  prototyping  plan  for  each 
program  would  be  based  primarily  on  the  judgment  of  experienced  engineers 
and  managers.  We  endorse  authorizing  program  managers  to  update  and  revise 
the  prototyping  plan  as  the  program  moves  toward  Milestone  I.  The  first 
version  of  the  plan  would  be  prepared  before  issuance  of  the  Request  for 
Proposals  for  the  Demonstration  and  Validation  contract.  Later,  the  program 
managers  should  be  allowed  to  update  the  plan  based  on  contractor  input 
provided  in  response  to  the  Request  for  Proposals  and  ARPA  recommendations 
made  at  Milestone  I.  Accordingly,  we  request  that  the  Director,  Program 
Analysis  and  Evaluation,  reconsider  his  position  in  response  to  the  final  report. 

Appendix  I  contains  audit  responses  to  Director,  Program  Analysis  and 
Evaluation,  comments  on  the  text  of  our  finding. 
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Part  III  -  Additional  Information 


Appendix  A.  Use  of  Prototyping  in  the 

Acquisition  Process 


History  of  Prototyping.  Prototyping  has  long  been  part  of  the  acquisition 
process  for  weapon  systems.  However,  the  use  of  prototyping  has  been  cyclical 
according  to  RAND  National  Defense  Institute  Report  No.  R-4161-ACQ,  "The 
Nature  and  Role  of  Prototyping  in  Weapon  System  Development,"  1992. 
RAND  prepared  this  report  for  the  Under  Secretary  of  Defense  for  Acquisition 
and  Technology  (USD[A&T]). 

Early  Usage.  The  RAND  Report  states  that  prototyping  of  aircraft 
engine  and  airframe  combinations  was  the  pattern  of  aircraft  development  before 
1940  and  was  common  into  the  1950s.  With  the  advent  of  the  "total  system 
concept"  in  the  early  1950s,  the  use  of  prototyping  declined.  The  pattern 
reversed  in  the  late  1960s  when  (then)  Deputy  Secretoy  of  Defense  Dr.  David 
Packard  instituted  a  "fly  before  you  buy  policy."  While  the  use  of  prototyping 
was  included  in  DoD  acquisition  regulations  in  the  1970s,  the  RAND  Report 
states  that  the  Military  Departments  resisted  the  policy  and  it  was  not  fully 
accepted  or  applied. 

Renewed  Interest.  In  the  mid-1980s,  DoD  interest  in  prototyping  again 
increased.  In  June  1986,  "A  Quest  for  Excellence,  Final  Report  to  the 
President  by  the  President's  Blue  Ribbon  Commission  on  Defense 
Management,"  referred  to  as  the  Packard  Commission  Report  after  the  chairman 
of  the  Commission,  recommended  that  a  high  priority  be  given  to  building  and 
testing  prototype  systems  to: 

o  Demonstrate  that  new  technology  can  substantially  improve 
military  capability  and 

o  Provide  a  basis  for  realistic  cost  estimates  before  a  full-scale 
development  decision. 

The  Packard  Commission  emphasized  that  prototyping  before  commitment  to 
full-scale  development  (currently  Phase  II,  Engineering  and  Manufacturing 
Development)  would  enhance  source  selection,  provide  early  demonstrations  of 
the  feasibility  and  operational  utility  of  new  technologies,  and  improve  initial 
cost  estimates  of  new  systems.  Subsequent  studies  by  RAND,  the  Institute  for 
Defense  Analysis,  and  the  Defense  Science  Board  also  advocated  greater  use  of 
prototyping. 

When  the  Packard  Commission  Report  was  issued,  both  Congress  and  DoD 
began  to  encourage  use  of  prototyping.  In  1986,  Congress  added  title  10, 
United  States  Code,  section  2365,  "Competitive  prototype  strategy  requirement; 
major  Defense  acquisition  programs,"  which  expired  on  September  30,  1991. 
The  statute  required  that  before  development  under  an  MDAP  begins,  contracts 
must  be  entered  into  with  two  or  more  contractors  for  competitive  design  and 
manufacture  of  a  prototype  system  or  subsystem.  Only  Congress  could  waive 
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the  requirement.  The  1991  expiration  date  was  designed  to  permit  DoD  time  to 
develop  appropriate  regulatory  guidance  for  use  of  competitive  prototyping 
strategies.  On  October  23,  1992,  the  competitive  prototyping  requirement  was 
reinstated  in  title  10,  United  States  Code,  section  2438,  "Major  Programs: 
Competitive  Prototyping."  Reinstatement  was  prompted  by  congressional 
observation  of  the  success  of  a  program  that  used  competitive  prototyping  and 
problems  in  programs  that  had  not. 

DoD  acquisition  regulations  became  more  specific  on  the  requirements  for  using 
prototyping  with  the  issuance  of  DoD  Instruction  5000.2,  "Defense  Acquisition 
Program  Procedures,"  September  1,  1987.  This  version  of  the  Instruction 
required  establishment  of  a  competitive  prototyping  strategy  at  Milestone  I, 
Concept  Demonstration/Validation  Decision  (now  Concept  Demonstration 
Approval),  and  use  of  results  from  prototyping  to  support  the  Milestone  II, 
Full-Scale  Development  Decision  (now  Development  Approval).  Prototyping 
has  remained  a  part  of  the  Instruction  through  all  subsequent  updates. 

Recent  Initiatives.  In  January  1993,  the  report  of  the  Acquisition  Law 
Advisory  Panel  to  the  United  States  Congress,  "Streamlining  Defense 
Acquisition  Laws,"  recommended  that  title  10,  United  States  Code,  section 
2438  be  repealed.  While  the  Panel  believed  that  competitive  prototyping  makes 
"eminently  good  sense,"  the  panel  did  not  believe  it  should  be  statutorily 
prescribed.  Rather,  the  Panel  believed  that  the  better  approach  is  to  allow  the 
Secretary  of  Defense  to  address  the  issue  by  internal  regulation  and  ensure  that 
competitive  prototyping  is  considered  as  part  of  the  overall  acquisition  strategy. 
The  Panel  concluded  that  DoD  has  regulations  that  encourage  competitive 
prototyping  in  certain  circumstances  and  that  repeal  of  the  statute  would  provide 
DoD  greater  management  flexibility  in  developing  its  acquisition  strategies.  On 
October  26,  1993,  Senate  Bill  S.1587  was  introduced  that,  if  passed,  will  repeal 
Section  2438. 

Increased  Use  of  Prototyping.  Continued  encouragement  of  the  use  of 
prototyping  by  Congress  and  DoD  has  increased  its  use  in  the  acquisition  of 
MDAPs.  Appendix  B  shows  that  prototyping  was  used  in  53  percent  of  the 
MDAPs  we  surveyed  that  started  since  1987  as  compared  with  40  percent  for 
MDAPs  started  before  1987.  However,  those  statistics  are  based  on  the 
traditional  hardware-oriented  definition  of  prototyping  that  does  not  include 
electronic  design  simulation  and  analysis,  now  widely  used  in  DoD  programs. 
As  discussed  in  this  report,  the  use  of  such  "electronic  prototyping"  should  be 
more  fully  considered  in  prototyping  acquisition  strategies.  Also,  we  used  the 
same  methodology  as  the  RAND  National  Defense  Institute  in  its  study  that 
credit^  programs  as  having  used  prototyping  if  it  was  used  to  any  extent 
regardless  of  whether  additional  prototyping  benefits  were  available  but  not 
pursued  for  the  programs. 
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Programs  Used  Some  Prototyping 


Program 

System 

Level 

Subsystem! 

Level 

Software 

Level 

Manufacturing 

Process 

AWACS  RSIP 

N 

Y 

Y 

N 

Comanche 

Y 

Y 

Y 

Y 

csscs 

N 

Y 

Y 

N 

FA-18  E/F 

N 

N 

N 

N 

FMTV 

N 

N 

N 

N 

JDAMS 

N 

Y 

Y 

Y 

JPATS^ 

Y 

N 

N 

N 

JSIPS 

N 

N 

N 

N 

JSOW^ 

Y 

Y 

N 

N 

Kiowa  Warrior 

N 

N 

N 

N 

PLS 

N 

N 

N 

N 

SCAMP 

Y 

Y 

N 

N 

SMART-T 

N 

N 

N 

N 

SRUAV 

N 

N 

N 

N 

UTTMDS 

Y 

Y 

Y 

Y 

Y  -  prototyping  used 
N  -  prototyping  not  used 


*We  assumed  that  some  partial  or  subsystem  level  prototypes  were  built  if  a  program  used 
system-level  prototyping.  We  also  credited  a  program  as  prototyping  at  the  subsystem 
level  if  any  aspect  of  the  program  was  prototype.  In  some  cases,  however,  further 
prototyping  opportunities  existed -on  other  subsystems. 


^Competing  contractors  did  some  degree  of  prototyping  before  contract  award  or  outside 
the  scope  of  the  contract. 
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Summary 

Of  the  126  Major  Defense  Acquisition  Programs  identified  in  the  most  recent  USD(A&T) 
listing  dated  July  19,  1993,  15  ongoing  programs  had  a  Milestone  I  review  since  1987. 
Our  analysis  excluded  ship  and  satellite  programs.  We  found  that  8  of  15  (53  percent) 
programs  used  or  plan  to  use  prototyping  strategies  before  EMD.  We  compared  our  audit 
results  to  those  of  the  IDA  November  1992  study  for  USD(A&T).  We  omitted  ship  and 
satellite  programs  from  the  results  the  IDA  reported  to  be  comparable  with  our  data  base. 
After  we  omitted  ship  and  satellite  programs,  we  found  that  37  of  93  (40  percent) 
programs  started  before  1987,  which  were  identified  over  32  years,  used  prototyping 
strategies. 


Legend 


AWACS  RSIP 

CSSCS 

FMTV 

JDAMS 

JPATS 

JSIPS 

PT  S 

SCAMP 

SMART-T 

SRUAV 


Airborne  Early  Warning  and  Control  System  Radar  System 
Improvement  Program 
Combat  Service  Support  Control  System 
Family  of  Medium  Tactical  Vehicles 
Joint  Direct  Attack  Munitions  System 
Joint  Primary  Aircraft  Training  System 
Joint  Service  Imagery  Processor 
Palletized  Load  System 

Single  Channel  Anti-Jam  Manportable  Terminal 
Secure  Mobile  Anti-Jam  Reliable  Tactical  Terminal 
Short  Range  Unmanned  Aerial  Vehicle 
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Concerning  the  Use  of  Prototyping 

Most  (four,  five,  or  six)  of  the  six  contractors  surveyed  agreed 
to  each  of  the  following  statements: 

-  Prototyping's  effectiveness  as  a  risk-reduction  tool  suffers  when  it  occurs 
outside  the  contract,  i.e.,  on  corporate  funds  (Independent  Research  and 
Development  or  Bid  and  Cost  Proposal). 

-  Government  funding  and  planning  must  remain  consistent  if  prototyping  is  to 
be  successful. 

-  Full-system  prototypes  are  good  marketing  tools  for  Congress,  but  are  not  as 
cost-effective  as  subsystem  prototyping  in  high-risk  areas. 

-  Critical  software  algorithms  should  be  prototyped. 

-  Properly  done,  prototyping  is  beneficial  to  a  program. 

-  A  written  Defense  Acquisition  Board-approved  prototyping  plan  will  help 
outline  contractor  and  program  office  responsibilities.  This  strategy  will  help 
preliminary  (Pre-milestone  I)  prototype  planning. 

-  Operational  Test  and  Evaluation  should  use  prototypes  to  the  maximum  extent 
practicable. 

-  Critical  manufacturing  technologies,  especially  important  materials,  should  be 
prototyped  as  early  as  possible  in  Demonstration  and  Validation. 

-  Models  and  simulations  should  be  used  extensively,  but  their  effectiveness  is 
limited  unless  prototyping  verifies  the  results. 

-  Prototyping  should  be  used  in  direct  proportion  to  the  level  of  expected 
technological  risk. 

-  Prototyping  Demonstration  and  Validation  should  be  done  using  a  cost-plus 
contract. 

-  ARPA  could  play  a  limited  advisory  role. 
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General  Accounting  Office 


Report  No.  GAO/NSIAD-90-107  (OSD  Case  No.  8341),  "Weapons  Testing: 
DoD  Needs  to  Plan  and  Conduct  More  Timely  Operational  Tests  and 
Evaluation,"  May  17,  1990,  found  that  although  some  prototypes  have  been 
used  for  Development  and  Test  and  Evaluation,  the  Military  Departments 
generally  are  not  planning  to  use  them  for  Operational  Test  and  Evaluation 
before  production  start-up.  The  General  Accounting  Office  believes,  however, 
that  with  adequate  planning,  prototypes  can  be  designated  for  Operational  Test 
and  Evaluation. 

Report  No.  GAO/NSIAD-89-98  (OSD  Case  No.  7800),  "Navy  Weapons 
Testing:  Defense  Policy  on  Early  Operational  Testing,"  May  8,  1989,  found 
that  the  Navy  often  proceeded  to  full-scale  development  or  low-rate  initial 
production  before  Operational  Test  and  Evaluation  was  complete.  The  General 
Accounting  Office  recommended  that  the  Secretary  of  Defense  reemphasize  the 
desirability  of  performing  Operational  Test  and  Evaluation  of  repr^entative 
hardware  as  early  as  possible  in  the  acquisition  cycle.  The  OSD  Director  of 
Operational  Test  and  Evaluation  partially  concurred  and  stated  that  a 
redefinition  of  the  testing  process  had  already  been  proposed  to  the  House  and 
Senate  Armed  Services  Committees.  The  OSD  proposal  would  enhance  the 
valuations  of  new  systems  by  providing  for  operationally  oriented  evaluations  of 
a  system's  expected  capability  as  early  as  possible  in  the  acquisition  process. 

Report  No.  GAO/NSIAD-89-72  (OSD  Case  No.  7844),  "Light  Helicopter 
Program:  Risks  Facing  the  Program  Raise  Doubts  About  the  Army's 
Acquisition  Strategy,"  December  23,  1988,  found  that  the  current  Light 
Helicopter  Program  Acquisition  Strategy  of  eliminating  test  and  evaluation  of 
prototypes  runs  counter  to  DoD  acquisition  guidance. 


Office  of  the  Inspector  General,  Department  of  Defense 


Report  No.  94-014,  "Low-Rate  Initial  Production  in  Major  Defense 
Acquisition  Programs,"  November  9,  1993,  recommended  establishing  a 
required  milestone  review  before  entry  into  Low-Rate  Initial  Production  (LRIP). 
We  believe  the  expanded  use  of  prototypes  will  be  consistent  with  improving  the 
quality  of  both  EMD  and  low-rate  initial  production  milestone  decisions. 
Report  recommendations  were  made  to  the  Under  Secretary  of  Defense  for 
Acquisition  and  as  follows: 
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1.  Revise  DoD  Instruction  5000.2  concerning  major  Defense  acquisition 
programs  to: 


a.  Establish  a  required  milestone  review  for  entry  into  low-rate 
initial  production,  including  confirmation  of  required  inidal  production 
quantities,  attainment  of  all  required  exit  criteria,  and  designation  of  approval 
authority  for  subsequent  low-rate  initial  production  contracts.  Program 
documentation  listed  in  Appendix  F  (of  Low-Rate  Initial  Production  report) 
should  be  required  for  the  milestone  review. 

b.  Provide  guidance  on  the  specific  minimum  required  program 
accomplishments  for  initially  committing  long-lead  procurement  funding  for 
low-rate  initial  production,  entering  low-rate  initial  production,  and  awarding 
subsequent  low-rate  initial  production  lots.  As  a  minimum,  operational 
assessment,  design  and  production  readiness  review,  and  operational  testing 
prerequisites  should  be  established. 

c.  Require  that  program-specific  exit  criteria  be  established  for 
initial  long-lead  procurement  funding,  entry  into  low-rate  initial  production,  and 
subsequent  initial  production  lots  at  the  Milestone  II  decision  point  and 
incorporated  as  events  in  development  contracts. 

d.  Direct  that  engineering  and  manufacturing  development 
contracts  include  requirements  for  production-representative  engineering 
development  models  for  purposes  of  performing  operational  assessments  before 
initially  committing  long-lead  procurement  funding,  unless  specifically  waived 
by  the  milestone  decision  authority  at  Milestone  II,  Development  Approval. 

2.  Require  Military  Standard  499A  to  include  a  direct  link  betw^n 
systems  engineering  requirements  and  low-rate  initial  and  full-rate  production 
decisions. 

3.  Require  that  the  minimum  low-rate  initial  production  quantities 
necessary  for  initial  operational  test  and  evaluation,  establishment  of  a 
production  base,  and  orderly  increase  to  a  full-rate  production  be  separately 
identified,  documented,  and  approved  by  the  milestone  decision  authority  at  the 
Milestone  II,  Developmental  Approval,  decision  point  and  reaffirmed  before 
entry  into  low-rate  initial  production. 

4.  Review  low-rate  initial  production  quantities  in  the  guidelines  for 
production  readiness  reviews  and  require  certification  as  a  result  of  the  review 
of  the  minimum  low-rate  initial  production  quantity. 

5.  Require  that  the  cost  and  benefits  of  a  break  in  production  versus 
annual  low-rate  initial  production  buys  be  assessed  by  the  milestone  decision 
authority  before  entry  into  low-rate  initial  production  to  limit  low-rate  initial 
production  quantities  to  the  minimum  necessary  while  providing  production 
units  for  operational  test  and  evaluation. 
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6.  Require  Service  Acquisition  and  Technology  Executives  to  request  a 
program-specific  waiver  from  the  Under  Secretary  of  Defense  for  Aojuisition 
before  award  of  low-rate  initial  production  contracts  whenever  testing  and 
review  prerequisites  in  DoD  regulations  and  the  approved  acquisition  strategy 
are  deleted,  modified,  or  not  met. 

As  of  March  11,  1994,  the  Under  Secret^  of  Defense  for  Acquisition  and 
Technology  had  tentatively  planned  corrective  actions  that  meet  the  intent  of  all 
recommendations.  On  May  11,  1994,  the  Assistant  Inspector  General  for 
Auditing  agreed  with  a  proposal  from  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology  to  implement  the  recommendations  together  with 
recommendations  to  be  presented  by  the  Defense  Acquisition  Board  Process 
Action  Team. 

Report  No.  93-017,  "The  Critical  Design  Review  Process  for  Major  Defense 
Acquisition  Programs,"  November  5,  1992,  recommended  that  the  Critical 
Design  Review  be  an  element  of  the  acquisition  program  baseline  and  that 
completion  of  the  Review  be  an  exit  criteria  for  entering  production.  We 
believe  that  a  significant  level  of  design  review  can  be  supported  through  the 
use  of  prototypes.  The  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  agreed  to  modify  DoD  Instruction  5000.2  to  require  that  the 
completion  date  for  the  Critical  Design  Review  be  included  in  the  Acquisition 
Program  Baseline  and  to  require  that  the  Critical  Design  Review  be  included  as 
one  of  the  minimum  requirements  for  approval  of  entry  into  low-rate  initial 
production. 

Report  No.  91-INS-05,  "Defense  Advanced  Research  Projects  Agency," 
March  15,  1991,  determined  that  no  formal  process  provides  structure  to  the 
Defense  Advance  Research  Projects  Agency's  (now  ARPA)  involvement  in 
prototyping.  No  official  policy  addresses  prioritization,  approval,  funding, 
oversight,  and  other  issues  relating  to  the  ARPA’s  expand^  prototype  role. 
The  Report  recommended  that  the  Under  Secretai7  of  Defense  for  Acquisition 
(now  USD[A&T])  clarify  the  ARPA  prototype  mission  and  establish  a  process 
governing  ARPA  involvement  in  prototyping.  The  Under  Secretary  of  Defense 
for  Acquisition  partially  concurr^  with  the  recommendations  stating  that  the 
Agency’s  mission  has  been  clarified  by  the  Executive  Director  of  the  Packard 
Commission  in  response  to  a  request  by  the  Director,  Advanced  Research 
Projects  Agency,  to  do  so  (the  "clarification"  refers  to  the  April  1989  letter 
from  Dr.  David  Packard  and  Dr.  William  J.  Perry  discussed  under  "ARPA 
Involvement  in  Formulating  Prototyping  Strategies").  The  Under  Secretary  of 
Defense  for  Acquisition  further  stated  that  the  Secretary  of  Defense  can  task  the 
ARPA  to  perform  such  technology  projects  as  deem^  necessary  and  that  no 
additional  processes  for  governing  the  Agency's  involvement  in  prototyping 
were  needed. 


55 


Appendix  D.  Prior  Audits  and  Other  Reviews 


Other  Reviews 


"Report  of  the  Defense  Science  Board  Task  Force  on  Aircraft  Assessment," 
February  25,  1993,  stated  that  both  ground  and  flight  prototype  testing  during 
Phase  I,  Demonstration  and  Validation,  reduced  the  technical  risk  of  aircraft 
programs,  thereby  reducing  the  schedule  and  cost  risks  in  proceeding  to  HMD 
and  production. 

RAND  Report  R-4161-ACQ,  "The  Nature  and  Role  of  Prototyping  in 
Weapon  Systems  Development,"  1992,  concluded  that  some  form  of 
prototyping  is  almost  always  appropriate.  RAND  indicated  that  prototyping  will 
generate  information  to  improve  the  quality  of  decisionmaking  in  an 
environment  of  risk  and  uncertainty.  The  Report  recommended  that  a  rational 
decisionmaking  process  for  considering  the  full  range  of  prototyping 
considerations  be  defined  and  applied  to  the  development  of  acquisition 
strategies  for  weapon  systems. 

IDA  Paper  P-2722,  "The  Effects  of  Management  Initiatives  on  the  Costs 
and  Schedules  of  Defense  Acquisition  Programs,"  November  1992,  states  that 
prototyping,  when  done  before  full-sc^e  development,  greatly  reduced 
development  cost  growth  in  tactical  munition  programs,  where  the  programs 
with  the  greatest  technical  risk  were  prototyped.  In  addition,  prototyping  helps 
developers  and  users  understand  technics  risks  and  uncertainty  of  the 
requirements.  The  Paper  recommended  that  acquisition  managers  consider  using 
more  prototyping. 

The  Institute  for  Defense  Analyses  Document  D-1097,  "Prototyping  Defense 
Systems,"  December  1991,  concluded  that  prototyping  makes  development 
costs  more  predictable  and  production  cost  growth  lower.  The  Report 
recommended  prototyping  systems  involving:  new  performance  or 

manufacturing  technologies  for  the  contractor(s),  high  cost  per  unit  and  large 
production  quantities,  or  long  lead  time  or  high  cost  to  correct  potential 
unforeseen  problems. 

The  "Defense  Management  Report  to  the  President,"  June  12,  1989, 
recommended  a  more  vigorous  policy  direction  in  the  early  phases  of  system 
development,  especially  those  that  involve  funds  and  schedules  for  prototyping 
and  testing.  The  report  advocated  building  and  testing  system  and  critical 
subsystem  prototypes  as  well  as  the  early  test  and  evaluation  of  prototype 
hardware. 

"Packard  Commission  Report,"  June  1986,  recommended  that  the  role  of 
Defense  Advance  Research  Projects  Agency  (now  ARP  A)  should  be  expanded 
to  include  the  additional  mission  of  stimulating  greater  emphasis  on  prototyping 
Defense  systems.  ARPA  should  actually  conduct  prototype  projects  that 
embody  technology  that  might  be  incorporated  in  joint  or  selected  Military 
Department  programs.  ARPA  should  also,  upon  request,  assist  the  Military 
Departments  in  their  own  prototyping  programs.  The  Packard  Commission  also 
strongly  encouraged  prototyping  to  resolve  program  risk. 
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DoD  Instruction  5000.2,  "Defense  Acquisition  Policies  and  Procedures": 
The  Instruction,  issued  February  23,  1991,  provides  policies  and  procedures  for 
program  managers,  milestone  decision  authorities,  and  the  respective  supporting 
staffs  to  acquire  products  that  emphasize  acquisition  planning,  communications 
with  users,  and  risk  management  in  all  phases  of  the  acquisition  process.  The 
Instruction  identifies  prototyping  as  a  major  element  of  Phase  I,  Demonstration 
and  Validation,  to  assess  and  reduce  risks  from  integrating  available  ^d 
emerging  technologies  to  satisfy  a  valid  mission.  In  addition,  the  Instruction 
provides  the  following  guidance: 

0  Programs  will  establish  requirements  for  prototyping  at  Milestone  I, 
Concept  Demonstration  Approval.  Those  requirements  will  be  based  on  an 
assessment  of  technologic^,  manufacturing,  and  cost  risk  associated  with  the 
proposed  concept  and  the  results  of  technology  demonstrations. 

o  Prototyping  is  a  critical  element  in  meeting  the  objectives  for  Phase 
I,  Demonstration  and  Validation,  to  reduce  risk  and  provide  the  operational  test 
organization  an  opportunity  for  early  operational  assessment  of  effectiveness  and 
suitability.  Prototyping  will  also  provide  the  opportunity  for  early  assessment  to 
identify  the  need  for  new  or  modified  test  capabilities  to  support  system 
development. 

o  Programs  should  use  the  results  of  the  test  and  evaluation  of 
prototypes  to  confirm  the  feasibility  of  a  specific  design  approach  relative  to  its 
ability  to  satisfy  the  mission  need  and  to  achieve  acceptable  operational 
performance  requirements  within  affordable  constraints. 

o  Programs  should  establish  a  competitive  prototyping  strategy  at 
Milestone  I,  Concept  Demonstration  Approval,  and  use  the  results  to  support 
the  Milestone  II,  Development  Approval.  Programs  are  to  use  competitive 
prototyping  of  systems  or  critical  subsystems  unless  the  milestone  decision 
authority  issues  a  specific  waiver. 

0  Programs  may  not  enter  Phase  II,  EMD,  unless  the  milestone 
decision  authority  confirms,  before  the  Milestone  II,  Development  Approval, 
that  prototyping  results  provide  reasonable  assurance  that  the  technologies  and 
processes  critical  to  success  are  attainable. 

DoD  4245.7-M,  "Transition  from  Development  to  Production":  The 
Manual,  issued  in  September  1985,  helps  structure  technically  sound  programs, 
assess  their  risk,  and  identify  areas  needing  corrective  action.  The  assistance  is 
in  a  series  of  templates;  each  template  discusses  an  area  of  risk  and  then 
provides  methods  for  reducing  that  risk.  The  templates  are  based  on  lessons 
learned  from  analyses  of  programs. 

The  Manual  states  that  program  risk  is  introduced  when  a  particular  element  is 
started  late  or  continues  beyond  the  time  line.  Knowledge  gained  through 
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prototyping  experiences  during  Demonstration  and  Validation  can  minimize  risk 
for  30  of  31  elements  (97  percent)  identified  under  "Design,"  "Testing,"  and 
"Production"  categories  list^  in  the  Manual. 

Military  Standard  2167A,  "Defense  System  Software  Development":  The 
Military  Standard,  issued  February  29,  1988,  establishes  uniform  software 
requirements  that  are  applicable  throughout  the  system's  life  cycle.  The 
requirements  of  the  standard  allow  the  Government  to  oversee  a  contractor's 
software  development,  testing,  and  evaluation  efforts.  The  requirements  in  the 
standard  apply  to  the  development  of  computer  software  configuration  items. 

The  standard  must  be  appropriately  tailored  to  require  only  what  is  needed  for 
each  acquisition.  The  program  manager  should  tailor  this  standard  to  ensure 
that  only  cost-effective  requirements  are  cited  in  Defense  solicitations  and  new 
contracts.  Specific  tailoring  guidance  is  in  DoD  Handbook  248,  "Guide  for 
Application  and  Tailoring  of  Requirements  for  Defense  Material  Acquisitions." 

Draft  Military  Standard  498,  "Software  Development  and  Documentation," 
December  22,  1992,  will  merge  DoD  Standard  2 167 A,  "Defense  System 
Software  Development";  DoD  Standard  7935A,  "DoD  Automated  Information 
Systems  Documentation  Standards";  and  the  Data  Item  Descriptions  associated 
with  those  standards.  Thus,  a  single  DoD  software  development  standard  will 
be  usable  for  any  software.  The  standard  will  establish  uniform  requirements 
for  software  development  that  are  applicable  throughout  the  system's  life  cycle. 
As  required  for  application  of  Military  Standard  2176A,  this  standard  also 
provides  for  appropriate  tailoring  to  require  only  what  is  needed  for  each  item 
or  type  of  software  developed. 

Military  Standard  1521-B,  "Technical  Reviews  and  Audits  for  Systems, 
Equipments,  and  Computer  Software":  This  military  standard,  issued 
June  4,  1985,  identifies  requirements  for  technical  reviews  and  audits,  which 
occur  throughout  the  acquisition  process.  The  specific  reviews  that  occur 
during  Phase  I,  Demonstration  and  Validation  include: 

o  System  Design  Review:  This  review  is  conducted  to  evaluate  the 
optimization,  correlation,  completeness,  and  risks  associated  with  the  allocated 
technical  requirements.  This  review  ^so  includes  a  summary  review  of  the 
system  engineering  process  and  engineering  plans  for  the  next  phase  of  the 
effort.  This  review  is  conducted  when  the  system  definition  effort  has 
proceeded  to  the  point  where  system  characteristics  are  defined  and  the 
configuration  items  are  identified. 

0  Preliminary  Design  Review:  This  review  is  a  formal  technical 
review  of  the  basic  design  approach  for  a  configuration  item  or  aggregate  of 
configuration  items.  TTie  overall  technical  program  risks  associated  with  each 
configuration  item  shall  be  reviewed  on  a  technical,  cost,  and  schedule  basis. 
For  computer  software  configuration  items,  this  review  will  focus  on: 

o  the  evaluation  of  the  progress,  consistency,  and  technical 
adequacy  of  the  selected  top-level  design  and  test  approach; 
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o  compatibility  between  software  requirements  and  preliminary 

design;  and 

0  the  preliminary  version  of  the  operation  and  support 

documents. 

Draft  Military  Standard  499B,  "Systems  Engineering":  The  draft  military 
standard,  dat^  May  6,  1992,  is  being  prepared  by  OSD,  the  Mili^ 
Departments,  and  an  Industry  Steering  Committee.  The  draft  standard  requires 
establishing  and  implementing  a  structured,  disciplined,  and  documented 
systems  engineering  effort  incorporating  the  systems  engineering  process;  multi¬ 
disciplinary  teamwork;  and  the  simultaneous  development  of  products  and 
processes  needed  to  satisfy  operational  needs.  The  draft  standard  defines  a  total 
system  approach  for  developing  Defense  systems,  defines  requirements  for 
technical  reviews,  and  provides  a  method  for  evaluating  progress  in  achieving 
system  objectives.  In  regard  to  prototyping,  section  5.4.3,  "Prototyping," 
requires  that  organizations  evaluate  whether  prototyping  should  be  used  to 
identify  and  reduce  risk  of  integrating  available  and  emerging  technologies  into 
an  item's  design  for  satisfying  requirements.  Prototyping  should  provide  timely 
assessment  of  item  testability  to  identify  the  need  for  new  or  modified  test 
capabilities.  In  addition,  prototyping  should  address  all  aspects  of  the  emerging 
technology  that  bear  upon  its  successful  application  (i.e.,  software,  hardware, 
and  manufacturing  processes). 
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Italic  sections  show  proposed  changes  to  DoD  Instruction  5000>2. 


MILESTONE 

I 

MILESTONE 

iliiiiMa  '■! 

MILESTONE 

IM 

MILESTONE 

ra 

•  Approval  to 
enter  Phase  I 

•  Approve  the 
acquisition 
strategy  (with 
prototyping  plan) 
and  concept 
baseline  (with 
prototyping 
milestone  dates) 

•  Establish  exit 
criteria  for  Phase 

I  (to  include 
quantifiable  exit 
criteria  for  the 
use  of  prototypes) 

•  Require 
independent  test 
organizations 
report  on 
acquisition 
strategy  reports 
(to  include 
prototyping  plans 
relative  to 
performance  of 
operational 
assessments) 

•  Require  ARPA 
and  OSD  to 
report  on 
prototyping  plans 

•  Approval  to  enter 

Phase  II  based  on  meeting 
Phase  I  exit  criteria 
including  producibility 
analysis  based  on 
prototyping  and 
completing  an  initial 
design  review 

•  Approve  the  acquisition 
strategy  and  development 
baseline 

•  Establish  Milestone 

IIIA  review  for^rograms 
planning  LRIP^ 

•  Establish  specific  exit 
criteria  for  award  of  long- 
lead  procurement  for 

LRIF  and  for  Phase  II 

•  Identity  LRIP  quantities 
(Separate  LRIP  quantities 
approved  for  operational 
testing,  production  base, 
and  increase  to  full-rate 
production) 

•  Require  EMD  contracts 
to  provide  for  production- 
representative  models 
before  committing  long- 
lead  procurement  funding 

•  Approval  to  enter  Phase 
IIIA  ^ased  on,  at  a 
minimum,  operational 
assessment,  design  and 
production  readiness 
reviews,  demonstration  of 
operational  requirements, 
and  meeting  prograwr 
specific  exit  criteria) 

•  Reaffirm  LRIP 
quantities^ 

•  Update  the  acquisition 
strategy  report  and  ^ 
development  baseline^ 

•  Establish  program- 
specific  exit  criteria  for 
awoM  of  subsequent  LRIP 
lots^ 

•  Approval  to 
enter  Phase  III 

•  Approve  the 
acquisition 
strategy  and 
production 
baseline 

•  Establish 
specific  exit 
criteria  for 

Phase  III 

^Milestone  IIIA  will  be  an  additional  review  for  programs  that  the  Milestone  II  review 
determined  Low-Rate  Initial  Production  to  be  necessary  before  full  production. 

^Tentative  agreement  already  reached  with  USD(A&T)  to  implement  these  changes. 
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BAT.TSOWFDS  F-22  UTTMDS 


-  Prototyping  too  limited 
to  adequately  reduce  risk 

—  hardware  X  XXX 

—  software  X  X 

--  manufacturing  process  X  X 

-  Competition 

—  Waivers  required  but  X 

not  submitted  for  relief  from 

competitive  prototyping 
requirements 

—  Inadequate  control  over  X 

competitive  prototyping 

-  Acquisition  program  2  2  2  X 

baseline  for  Demonstration 

and  Validation  inadequate 
to  measure  progress  in 
prototyping 

Teeing 

-  Limited  Operational  Assessment 

Over-reliance  on  modeling  X 

and  simulation 

Inadequate  assessment  X  XX 

of  operational  suitability 

-  Planning  for  test  and  evaluation  XX  X 

of  software  insufficiently 

documented  at  transition  to  EMD 

'pertains  to  baseline  information  on  the  pre-planned  product  improvement  portion  of 
the  program,  which  is  still  in  Demonstration  and  Validation. 

^We  did  not  examine  Demonstration  and  Validation  baselines  for  these  programs  since 
they  had  already  progressed  to  EMD. 
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Hardware 

System 

Hardware 

Subsystem 

Software 

Manufacturing 

Program 

Level 

Level 

Level 

Process 

BAT 

N 

Y 

N 

N 

Comanche 

Y 

Y 

Y 

Y 

F-22 

Y 

Y 

Y 

Y 

FDS 

N 

Y 

N 

N 

JSOW* 

Y 

Y 

N 

N 

UTTMDS 

Y  -  prototyping  used 

N  -  prototyping  not  used 

Y 

Y 

Y 

Y 

^Competing  contractors  did  prototyping  outside  the  scope  of  the  contract. 
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The  Director,  Acquisition  Program  Integration;  the  Director,  Program  Analysis 
and  Evaluation;  and  the  Director,  Defense  Procurement,  provided  specific 
comments  on  the  body  of  the  audit  finding  in  the  draft  report.  Below  we 
discuss  each  management  comment  and  provide  our  audit  response. 


Director,  Acquisition  Program  Integration,  Comments 


Comments.  The  Director  questioned  why  we  did  not  discuss  the  Navy's  FDS 
in  the  body  of  the  draft  report.  The  FDS  is  one  of  the  four  programs  identified 
as  having  limitations  in  planning  or  execution  of  hardware  prototyping 
strategies. 

Audit  Response.  We  used  specific  program  examples  to  show  the  types  of 
limitations  identified  with  program  hardware  prototyping  strategies  rather  than 
discuss  in  detail  all  four  programs  we  identifi^  with  limitations.  Although  the 
FDS  system  was  not  used  as  an  example  in  our  discussion  of  hardware 
prototyping  strategies,  we  subsequently  did  discuss  limitations  in  the  FDS 
hardware  prototyping  strategy  as  part  of  our  discussion  of  "Early  Operational 
Assessments  of  Prototyping  Efforts. " 

Comments.  The  Director  stated  that  the  draft  report  was  incorrect  in  stating 
that  the  JSOW  program  did  not  plan  to  use  prototyping  in  Demonstration  and 
Validation.  Specifically,  the  JSOW  program  office  planned  an  appropriate 
amount  of  component  prototyping  using  the  Recoverable  Test  Vehicle  Program. 
When  the  Recoverable  Test  VeWcle  Program  ran  into  difficulties,  the  JSOW 
program  office  coordinated  with  the  three  Demonstration  and  Validation 
contractors  to  modify  their  plans  to  include  prototyping.  These  contractors  used 
their  own  funds  to  develop  and  demonstrate  the  maturity  of  their  systems.  If  the 
JSOW  program  office  had  contracted  for  three  independent  prototyping  efforts, 
the  cost  and  schedule  impact  on  the  program  would  have  been  an  additional 
$64  million  and  a  delay  of  2  years. 

Audit  Response.  We  agree  that  the  JSOW  Program  did  plan  to  use  prototyping 
in  Demonstration  and  Validation  as  discussed  in  the  draft  report.  However, 
what  we  pointed  out  in  the  draft  report  was  that  the  planning  and  execution  of 
prototyping  had  significant  limitations  for  the  JSOW  Program,  not  that  the 
Program  did  not  plan  to  use  prototyping  in  Demonstration  and  V^idation.  Our 
estimate  of  $34  million  to  modify  the  three  Demonstration  and  Validation 
contracts  to  require  competitive  prototyping  was  based  on  information  provided 
by  the  JSOW  Program  Office. 

Comments.  The  Director  stated  that  the  report  recognized  that  prototyping  is 
primarily  a  risk-reduction  activity  but  fails  to  recognized  that  the  whole  premise 


63 


Appendix  I.  Audit  Response  to  Specific  Management  Comments 


of  the  JSOW  program  is  low-risk  integration  of  existing  technology.  The 
Director  further  stated  that  experience  program  managers  and  engineers 
assessed  program  risks  and  developed  the  prototyping  strategy  accordingly.  It 
was  determined  at  Milestone  I  that  conducting  subsystem  testing,  analysis, 
simulations,  and  demonstrations  in  Demonstration  and  Validation  would  be 
adequate  to  support  a  decision  to  move  into  the  EMD  Phase. 

Audit  Response.  Development  of  JSOW  was  considered  low  risk  and  involves 
the  integration  of  existing  technology.  However,  the  point  we  were  conveying 
was  that  performing  the  vast  majority  of  prototyping  activity  outside  the  scope 
of  the  contracts  seriously  impact^  the  JSOW  Program  Office's  ability  to  further 
reduce  JSOW  development  risk  in  Demonstration  and  Validation  before 
proceeding  into  EMD.  The  JSOW  Program  Office’s  ability  was  impaired 
because  the  Program  office: 

o  could  not  directly  interface  with  the  contractors  concerning  how 
prototyping  could  be  used  most  effectively  and  efficiently; 

o  did  not  have  approval  authority  over  contractor  testing  of  prototype 
hardware;  and 

0  could  not  establish  exit  criteria  involving  events  or  achievements  using 
prototypes  to  support  Milestone  II,  Development  Approval,  and  award  of  the 
EMD  contract. 

Comments.  The  Director  stated  that  our  reference  to  "Shore  Signal 
Information  Processing  System"  should  read  "Shore  Signal  Information 
Processing  Segment." 

Audit  Response.  We  revised  the  final  report  as  suggested. 

Comments.  The  Director  stated  that  prototyping  of  software  was  not 
performed  for  the  underwater  segment  of  FDS  because  only  2,500  lines  of  code 
were  in  the  underwater  segment  of  FDS.  Therefore,  the  level  of  risk  did  not 
warrant  the  added  cost  of  prototyping  software. 

Audit  Response.  We  revised  the  first  sentence  of  our  FDS  example  in  the 
Software  section  of  the  final  report  to  read,  "Prototype  software  was  not 
developed  and  tested  during  Phase  I,  Demonstration  and  Validation  for  the 
shore  segment  of  the  FDS."  This  change  makes  it  clear  that  we  are  talking 
about  the  estimated  1.6  million  lines  of  c^e  for  the  shore  segment  of  FDS  that 
were  needal  to  test  the  hardware  for  the  underwater  segment  of  FDS  and  not 
the  specific  lines  of  code  for  the  underwater  segment  of  FDS.  The  shore 
segment  software  was  not  available  to  support  testing  of  the  underwater  segment 
hardware  because  of  the  29-month  disconnect  between  the  development  of  the 
underwater  segment  and  the  development  of  the  shore  segment.  The  underwater 
segment  was  approved  to  enter  EMD  in  September  1989  while  the  shore 
segment  did  not  enter  EMD  until  February  1992. 

Comments.  The  Director  stated  that  the  report  does  not  recognize  that  formal 
prototyping  was  not  accomplished  because  insufficient  funds  were  available. 
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Further,  the  FDS  program  office  believes  that  manufacturing  technology  during 
Demonstration  and  Validation  of  FDS  was  in  accordance  with  DoD  Directive 
4245.6. 

Audit  Response.  We  do  not  understand  management's  response  that  "formal 
prototyping  was  not  accomplished  because  insufficient  funds  were  available." 
The  IDA  Report  D-1097,  "Prototyping  Defense  Systems,"  December  1991, 
stated  that  prototype  cost  should  probably  be  less  than  25  percent  of  the  HMD 
cost  estimate.  During  Demonstration  and  Validation,  the  FDS  Program  Office 
received  approval  for  accelerated  funding  in  December  1982  (FY  1983),  and 
did,  in  fact,  expend  20  percent  ($249  million  of  $1.2  billion)  of  its  Research, 
Development,  Test  and  Evaluation  funds  from  FYs  1984  through  1989  during 
the  Demonstration  and  Validation  Phase.  Although  the  percentage  of  the  funds 
to  be  spent  on  prototyping  has  not  been  established,  20  percent  is  a  significant 
amount  of  money  to  be  spent  during  Demonstration  and  Validation  without 
formal  prototyping  of  manufacturing  processes. 

Also,  the  FDS  program  office  did  not  meet  the  requirements  of  manufacturing 
technologies  in  DoD  Directive  4245.6,  "Defense  Production  Management," 
January  19,  1984  (cancelled  by  DOD  Instruction  5000.2  in  1991).  The 
Directive  required  that  manufacturing  voids  and  deficiencies  be  addressed 
during  Demonstration  and  Validation  through  the  use  of  manufacturing 
technology  projects.  These  projects  are  to  involve  the  development  or 
improvement  of  manufacturing  processes  and  equipment  to  help  bridge  the  gap 
between  system  feasibility  and  production.  The  FDS  Program  Office  did  not 
use  manufacturing  technology  prototyping  during  Demonstration  and  Validation 
because  it  was: 

0  concentrating  on  verifying  the  feasibility  of  a  new  pattern  for 
distributing  listening  devices  by  using  existing  hardware  from  the  Sound 
Surveillance  System  (the  predecessor  to  FDS)  during  Demonstration  and 
Validation  and 

o  planning  for  the  development  of  hardware  to  be  accomplished  during 

EMD. 

Comments.  The  Director  disagreed  with  the  report  statement  that  the  JSOW 
program  flailed  to  comply  with  title  10,  United  States  Code,  section  2365,  which 
required  a  waiver  if  prototyping  was  not  used.  The  Director  stated  that  the 
JSOW  program  office  maintain^  that  it  was  conducting  an  appropriate  amount 
of  prototyping.  In  addition,  the  Under  Secret^  of  Defense  for  Acquisition  and 
Technology  stated  that  the  JSOW  prototyping  strategy  complied  with  DoD 
policy  at  Milestone  I,  Concept  Demonstration  Approval. 

Audit  Response.  We  stand  by  our  position  that  the  Navy  did  not  satisfy  the 
requirements  of  title  10,  United  States  Code,  section  2365  because  a  prototyping 
waiver  request  was  not  forwarded  through  the  Secretary  of  Defense  to  Congress 
as  required  in  1988.  As  stated  in  the  draft  report,  the  JSOW  acquisition  plan 
submitted  at  Milestone  I  did  not  require  the  contractors  to  use  prototypes  during 
Demonstration  and  Validation.  The  acquisition  plan  clearly  stated  that 
statements  of  work  for  the  Demonstration  and  Validation  contracts  would  stop 
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short  of  requiring  prototype  design,  fabrication,  and  test.  Therefore,  Congress 
should  have  been  informed  that  the  JSOW  Program  Office  did  not  intend  to  use 
a  competitive  prototype  acquisition  strategy  during  Demonstration  and 
Validation  as  required  by  title  10,  United  States  Code,  section  2365. 

Comments.  The  Director  disagreed  with  the  audit  conclusion  that  the 
Government  should  control  the  subcontract  competition  or  subject  the  prime 
contractor's  selection  of  a  subcontractor  to  Government  approval.  He  stated 
that  this  procedure  is  inconsistent  with  holding  the  prime  contractor  responsible 
for  the  performance  of  the  total  integrated  system. 

Audit  Response.  We  agree  with  the  management  comments.  We,  therefore, 
deleted  the  audit  conclusion  from  the  final  report. 

Conunents.  The  Director  stated  that  Footnote  2  in  Appendix  B  (Appendix  A  in 
the  Draft  Report)  was  incorrect  in  stating  that  the  Joint  Primary  Aircraft 
Training  System's  contract  had  been  awarded. 

Audit  Response.  We  agree.  We  revised  Footnote  2  in  Appendix  B 
accordingly. 

Comments.  The  Director  stated  that  the  draft  report  incorrectly  stated  that  the 
UTTMDS  Acquisition  Program  Baseline  did  not  include  delivery  dates  for  the 
prototype  missiles.  Also,  he  stated  that  the  Theater  High  Altitude  Area  Defense 
provisional  performance  criteria  (the  equivalent  of  an  Acquisition  Program 
Baseline)  has  test  start  dates  and  missile  delivery  dates. 

Audit  Response.  We  agree.  We  eliminated  the  references  to  UTTMDS  missile 
delivery  dates  in  the  fin^  report. 

Comments.  The  Director  took  exception  to  our  characterization  of  existing 
guidance  in  DoD  Instruction  5000.2  by  stating  that: 

0  The  Instruction  shows  a  clear  preference  for  prototyping  md  a 
requirement  to  discuss  prototyping  in  addressing  the  recommend^  acquisition 
approach. 

0  The  Instruction's  guidance  is  not  limited  to  competitive  prototyping. 

o  Section  4-D  of  DoD  Manual  5(XX).2-M  requires  discussing  the 
requirement  to  prototype  in  the  Acquisition  Strategy  Report,  approval  of  the 
Acquisition  Strategy  Report  by  the  milestone  decision  authority,  and  justifying  a 
decision  not  to  prototype  in  the  Acquisition  Strategy  Report. 

Audit  Response.  We  agree  with  the  Director's  characterization  of 
prototyping  requirements  in  the  DoD  5000  series  of  directives.  However,  we 
still  believe  that  the  prototyping  guidance  in  DoD  Instruction  5000.2  can  be 
improved  in  the  areas  of  planning  and  execution  of  prototyping  acquisition 
strategies  to  assess  and  reduce  program  risk.  The  draft  report  included  specific 
improvements. 
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Comments.  The  Director  stated  that  he  found  essentially  no  discussion  of  the 
Comanche  program  in  the  finding  text.  Also,  he  was  concerned  about  the 
Comanche’s  so-called  "streamlin^"  program  of  two  EMD  prototypes  and 
three  Low-Rate  Initial  Production  helicopters  rather  than  three  Demonstration 
and  Validation  prototypes  and  three  EMD  prototypes.  He  felt  that  a  great  deal 
of  risk  is  being  placed  on  the  Low-Rate  Initial  Production  decision  because  the 
Army  is  not  performing  final  development  testing  until  the  Low-Rate  Initial 
Production  helicopters  are  available  for  testing. 

Audit  Response.  We  agree  with  management  comments.  In  Appendix  F  of  the 
draft  report,  we  indicate  that  prototyping  was  too  limited  to  adequately  reduce 
hardware  risk  for  the  Comanche  Program  based  on  the  deferral  of  a  number  of 
Demonstration  and  Validation  tasks  to  EMD.  Although  we  did  not  cite  the 
Comanche  Program  as  a  specific  example  in  the  Hardware  section  of  the  draft 
report,  we  believe  that  the  Comanche  Program,  like  the  BAT  Program,  was 
progressing  to  EMD  when  the  risk  reduction  strategy  for  Demonstration  and 
Validation  was  incomplete. 


Director,  Program  Analysis  and  Evaluation,  Comments 


Comments.  The  Director  cited  selected  quotations  from  the  RAND 
Report  R-4161-ACQ,  "The  Nature  and  Role  of  Prototyping  in  Weapon  Systems 
Development,"  1992,  that  he  believed  argue  against,  rather  than  for,  a  highly 
structured  prototyping  plan.  The  RAND  Report  stated: 

o  On  deciding  via  policy  whether  or  not  to  prototype:  "This  analysis 
suggests  that  attempts  to  define  such  a  policy  at  any  but  the  broadest  levels 
should  be  avoided  ....  Thus,  it  is  not  possible  or  even  desirable  to  develop  a 
set  of  firm  decision  rules." 

o  On  performing  the  cost  versus  benefits  analysis  of  prototyping: 

In  summary,  there  are  tremendous  difficulties  involved  in 
operationalizing  the  simple  concept  depicted  in  Figure  6. 1 
(cost  versus  benefit).  We  cannot  be  confident  that  we  can 
identify  all  possible  benefits  and  costs  ....  Even  if  we 
could  measure  all  costs  and  benefits,  we  do  not  know  how 
to  consistently  weigh  them  in  a  decision  process. 

o  On  trying  to  analyze  costs  versus  benefits  of  prototyping:  "In  the 
end,  there  is  no  substitute  for  informed  judgment  made  by  experienced 
managers  and  engineers." 

o  Regarding  DoD  regulations:  "This  conceptual  framework  for  policy 
and  decisionmaking  is  quite  similar  to  what  is  contained  in  existing  regulations." 


67 


Appendix  I.  Audit  Response  to  Specific  Management  Comments 


o  On  prototyping  versus  non-prototyping:  "However,  there  are  few 
significant  differences  between  prototyping  and  non-prototyping  programs  with 
respect  to  cost  growth,  total  actual  program  duration,  or  sch^ule  slip." 

Audit  Response.  Management  has  interpreted  the  above  quotations  out  of 
context.  The  conclusions  and  recommendations  in  our  draft  report  are 
consistent  with  the  conclusions  and  recommendations  made  in  the  RAND 
Report. 


o  Our  draft  report  was  fully  consistent  with  RAND's  position  that 
policy  regarding  prototyping  should  be  defined  only  at  the  broadest  level.  We 
stated  that  no  single  prototyping  strategy  fits  all  programs,  but  that  basic 
guidelines  are  needed  to  determine  when  prototyping  is  appropriate  and  what 
types  of  prototypes  would  be  effective  and  efficient  in  reducing  risk  in  a 
particular  program.  We  stated  that,  at  a  minimum,  policy  should  provide: 

o  general  definitions  of  system  level,  partial  system  level,  and 
subsystem  level  prototyping  and 

o  considerations  (not  strict  rules)  for  determining  the  appropriate 
level  of  prototyping  for  hardware,  software,  and  manufacturing  processes. 

While  RAND  advocates  defining  prototyping  policy  at  broad  levels  only,  the 
need  for  a  prototyping  plan  is  supplied  by  RAND's  conclusion:  "We  believe 
that  the  policy  should  be  to  use  some  form  of  prototyping  in  almost  every  case 
and  the  burden  of  proof  should  be  on  those  that  argue  that  a  prototyping  activity 
is  unnecessary  or  impractical."  The  formulation  process  that  would  be 
necessary  to  develop  a  prototyping  plan,  as  we  conceive  it,  would  help  satisfy 
the  "burden  of  proof  that  RAND  discussed  by  documenting  the  cost-benefit 
analysis  performed  to  support  decisions  to  use  a  particular  level  of  prototyping 
or  to  support  not  using  prototyping.  We  realize  that  the  prototyping  plan  should 
incorporate  the  best  judgment  of  experienced  engineers  and  managers. 
However,  we  also  believe  that  program  managers  should  be  required  to 
document  and  justify  the  extent  of  prototyping  planned  for  Phase  I, 
Demonstration  and  Validation. 

o  We  recognize  the  difficulty  to  identify  and  weigh  cost  versus  benefits 
of  prototyping  and  that  much  judgment  is  involved  in  this  process.  However, 
the  RAND  report  listed  some  factors  that  need  to  be  evaluated  when  considering 
prototyping.  None  of  our  recommendations  advocates  a  firm  set  of  decision 
rules  or  assigning  specific  weights  to  a  given  factor  for  every  program! .  Our 
finding  and  recommendations  instead  discuss  the  need  for  better  defining  ^d 
documenting  the  decisionmaking  process  related  to  formulation  of  a  prototyping 
strategy  for  a  given  program. 

0  The  RAND  report  stated  that  its  conceptual  framework  for 
prototyping  policy  and  decisionmaking  is  consistent  with  the  portion  of  the  DoD 
Instruction  5(X)0.2  that  discusses  establishment  of  competitive  prototyping 
strategies  based  on  program-specific  risk  assessments.  RAbp  further  stated 
that  the  current  regulation  applied  only  to  competitive  prototyping  and  not  other 
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forms  of  prototyping  activity.  RAND  believed  that  the  criteria  provided  for 
competitive  prototyping  could  be  expanded  in  the  regulation  to  cover  other 
forms  of  prototyping  activity. 

0  While  the  RAND  report  did  say  that  RAND  could  identify  few 
significant  differences  between  cost  and  schedule  results  of  prototyped  versus 
nonprototyped  systems,  the  report  also  concluded  that  the  effect  of  prototyping 
on  program  outcomes  is  ambiguous  due  to  the  effect  of  confounding  variables. 
According  to  RAND,  these  variables  included  such  factors  as  budget  stability, 
technical  difficulty  (comparisons  are  biased  if  only  the  most  technically 
challenging  programs  are  selected  for  prototyping),  and  the  ability  and 
willingness  of  decisionmakers  and  program  managers  to  use  the  improved 
information  generated  through  use  of  prototype^  lC\ND  concluded  that  some 
form  of  prototyping  is  almost  always  appropriate.  RAND's  conclusion  was 
based  on  the  existence  of  powerful  institutional  pressures  that  lead  to  systematic 
underestimation  of  program  risks. 

Comments.  The  Director  questioned  our  presentation  of  data  taken  from  the 
IDA  Report  D-1097,  "Prototyping  Defense  Systems,"  December  1991,  that 
showed  that  development  and  production  cost  growth  was  significantly  less  for 
programs  using  prototypes.  He  contended  that; 

o  The  report  did  not  assert  unequivocally  that  cost  growth  was  less  for 
prototyped  programs  since  the  report  stated  that  prototyping  was  significantly 
beneficial  for  munitions  programs  with  the  benefits  for  aircraft  being  less 
apparent. 

o  The  report  did  not  count  programs  that  were  modifications  to  existing 
systems  as  having  prototypes.  The  Director  believed  that  the  previously 
existing  systems  could  be  considered  as  prototypes. 

Audit  Response.  While  the  IDA  report  did  state  that  the  effect  of  prototyping 
was  smaller  for  aircraft  programs  than  for  munitions,  the  report  also  offered 
some  explanations  for  the  less  dramatic  (and  not  statistically  significant)  effects 
in  aircraft: 

0  Historically,  aircraft  systems  that  were  pushing  the  state  of  the  art  the 
least  (such  as  the  F-5E  and  the  F-16)  were  prototyped,  while  others  that  were 
more  technically  difficult  (such  as  the  F-14)  were  not.  In  munitions,  the  study 
states  the  opposite  and  more  logical  approach  was  used. 

0  Prototyped  aircraft  showed  significantly  lower  development  costs  than 
nonprototyped  aircraft  when  helicopters  were  removed  from  the  statistics 
because  helicopters  generally  had  higher  cost  growth  regardless  of  prototyping 
strategy. 

Finally,  the  Director's  argument  that  the  IDA  report  should  have  counted 
existing  systems  as  prototypes  for  subsequent  modifications  is  not  logical  unless 
the  acquisition  strategy  for  the  modification  efforts  specified  some  level  of 
prototyping  for  the  technology  improvements  being  added  to  the  existing 
systems. 
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Comments.  The  Director  stated  that  the  sample  of  programs  we  selected  for 
review  was  too  small  to  be  statistically  significant  and  too  narrowly  focused. 
He  believed  that  our  sample  selection  precluded  identifying  the  effects  of  not 
prototyping  since  all  programs  we  discussed  used  prototyping  and  no  program 
sampled  had  completed  EMD,  which  precluded  identifying  the  effects  of 
prototyping  on  total  EMD  cost  and  schedule. 

Audit  Response.  We  did  not  select  the  programs  reviewed  with  the  intention  of 
performing  a  statistical  sample  or  making  statistical  projections.  The  scope  and 
methodology  section  of  the  draft  report  stated  that  programs  were  judgmentally 
selected  and  were  intended  to  provide  examples  of  the  types  of  deficiencies  that 
can  occur  relating  to  planning  and  execution  of  prototyping  strategies. 

Regarding  identifying  the  effects  of  prototyping  versus  not  prototyping  on  cost 
and  schedule,  these  relationships  had  already  been  explored  in  the  two  broadly 
based  reports  issued  by  RAND  and  the  IDA.  The  RAND  report  was  supported 
by  information  gathered  on  287  programs  from  1960  through  1988.  The  IDA 
report  included  data  on  52  major  acquisition  programs,  including  aircraft, 
tactical  munitions,  and  satellites  over  32  years  ending  December  1987.  After 
close  examination  of  the  scope  and  methodology  of  these  reports  and  discussions 
with  their  authors,  we  found  no  need  to  duplicate  the  extensive  efforts  that  had 
already  been  made  to  document  the  effect  of  prototyping  on  cost  and  schedule. 


Director,  Defense  Procurement,  Comments 


Comments.  The  Director  nonconcurred  with  revising  the  Defense  Federal 
Acquisition  Regulation  Supplement  to  provide  solicitation  language  to  use  in 
requesting  contractor  input,  through  Requests  for  Proposals,  concerning 
proposed  prototyping  plans  and  to  allow  development  of  data  item  descriptions 
for  contractor  prototyping  plans  and  contractor  reports  documentating  the  results 
of  prototyping  risk  reduction  efforts.  She  stated  that  it  would  be  more 
appropriate  to  revise  DoD  Instruction  5000.2  language  on  prototyping  to  require 
that  the  solicitations  proposal  preparation  instructions  include  a  complete 
description  of  the  prototyping  plan  required  to  be  submitted  with  the  proposal. 
She  stated  that  this  type  of  solicitation  requirement  will  have  to  be  tailored  for 
each  program  and  is  not  susceptible  to  standardization.  She  stated  that  any 
"standard"  Defense  Federal  Acquisition  Regulation  Supplement  provision  would 
become  redundant  "boiler  plate"  and  that  the  real  information  about  what  was 
required  would  have  to  appear  in  the  proposal  preparation  instructions  anyway. 
Further,  the  Director  stat^  that  the  ^rtion  of  our  recommendation  regarding 
the  development  of  data  item  descriptions  for  contractor  prototyping  plans  and 
contractor  reports  should  be  redirected  to  the  Director,  Continuous  Acquisition 
and  Life-Cycle  Plans  and  Policy  Division,  who  reports  to  the  Office  of  the 
USD(A&T). 

The  complete  text  of  management  comments  is  in  Part  IV. 
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Audit  Response.  The  Director,  Defense  Procurement,  comments  were 
responsive  to  the  intent  of  our  recommendations.  As  a  result,  we  redirected 
Recommendation  3.  (renumbered  l.a.[ll])  to  the  Under  Secretary  of  Defense 
for  Acquisition  and  Technology  and  added  Recommendation  l.d.  in  the  final 
report. 

Comments.  The  Director  stated  that  she  agreed  that  Government  approval  of 
the  source-selection  plan  and  source-selection  criteria  should  be  used  when  the 
Government  has  a  critical  interest  in  the  selection  of  a  subcontractor  or 
component.  However,  she  also  believed  that  subjecting  the  prime  contractor's 
selection  of  a  subcontractor  to  Government  approval  or  veto  would  be 
inconsistent  with  holding  the  prime  contractor  responsible  for  the  performance 
of  the  total  integrated  system. 

Audit  Response.  We  agree  as  stated  in  our  response  to  Director,  Acquisition 
Program  Integration,  comments. 

Comments.  The  Director  stated  that  Footnote  2  in  Appendix  B  was  incorrect  in 
stating  that  the  Joint  Primary  Aircraft  Training  System's  contract  had  been 
awarded. 

Audit  Response.  We  agree.  The  footnote  was  revised  in  the  final  report. 


Auditor  Note:  Although  not  requested  to  respond  to  the  draft  report,  we 
received  comments  from  the  Army  Program  Executive  Office,  Tactical 
Missiles,  and  the  Program  Manager,  Comanche  Helicopter  Program.  Where 
deemed  appropriate,  we  revised  the  final  report  in  respionse  to  the  management 
comments. 
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Recommendation 

Reference  Description  of  Benefit 


Amount  and/or 
Type  of  Benefit 


l.a.(l) 

Internal  Control.  Will  provide 
consistent  application  of  prototyping 
in  the  acquisition  process. 

Nonmonetary. 

l.a.(2) 

Internal  Control.  Will  ensure  that 
program  managers  consider 
prototyping  strategies  to  reduce 
technic^,  cost,  and  schedule  risk  of 
transitioning  from  Demonstration 
and  Validation  to  EMD. 

Nonmonetary. 

l.a.(3) 

Internal  Control.  Will  ensure  that 
input  from  ARPA  on  prototyping 
plans  will  be  available  to  the 
acquisition  decision  authorities 
before  the  Milestone  I,  Concept 
Demonstration  Approval,  decision. 

Nonmonetary. 

l.a.(4) 

Internal  Control.  Will  ensure  that 
input  from  ARPA  concerning 
prototyping  as  a  risk  reduction 
method  will  be  available  to  the 
acquisition  decision  authorities 
before  the  Milestone  IV,  Major 
Modification  Approval,  decision. 

Nonmonetary. 

l.a.(5) 

Internal  Control.  Will  ensure  that 
Military  Department-independent 
test  organizations  report  on  early 
operational  effectiveness  and 
suitability. 

Nonmonetary. 
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Recommendation 

Reference 

l.a.(6) 

l.a.(7) 

l.a.(8) 

l.a.(9) 

l.a.(lO) 

l.a.(ll) 

l.b. 


Description  of  Benefit 


Internal  Control.  Will  ensure  that 
managers  have  an  adequate 
reference  point  for  measuring  and 
reporting  the  status  of  program 
implementation  and  that  baseline 
information  includes  prototyping 
milestone  dates. 

Internal  Control.  Will  ensure  that 
development  decisions  are  based  on 
specific  program  accomplishments. 

Internal  Control.  Will  ensure  that 
initial  design  reviews  are  completed 
before  a  Milestone  II,  Development 
Approval. 

Internal  Control.  Will  reduce  risks 
related  to  manufacturing  processes 
critical  to  successful  production  of 
weapon  systems. 


Internal  Control.  Will  ensure  that 
contracts  cite  cost-effective 
requirements  from  DoD  Standard 
2167  or  applicable  commercial 
standards. 

Internal  Control.  Will  ensure  that 
contractor  inputs  are  considered  in 
preparing  prototyping  plans. 

Internal  Control.  Will  provide 
guidance  for  program  managers  to 
formulate  prototyping  plans  before 
concept  demonstration  approval. 


Amount  and/or 
Type  of  Benefit 


Nonmonetary. 


Nonmonetary. 


Nonmonetary. 


Undeterminable. 
Amount  will  vary 
with  each  program 
and  benefits  will 
continue  as  programs 
define  producibility 
quantities. 

Undeterminable. 
Amount  will  vary 
with  each  program. 


Nonmonetary. 


Nonmonetary. 
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Recommendation 

Reference 

l.c.(l) 

l.c.(2) 

1. d. 

2. 


Description  of  Benefit 


Internal  Control.  Removal  of  a 
faulty  reference  in  DoD  Instruction 
5000.2  will  ensure  better 
implementation  of  the  provisions  of 
the  instruction. 

Compliance  with  DoD  Guidance. 
Will  require  program  managers  to 
include  software  test  and  evaluation 
plans  as  part  of  the  TEMP. 

Internal  Control.  Will  ensure  new 
Data  Item  descriptions  are 
developed  as  ne^ed  to  support 
submission  of  contractor 
prototyping  plans  and  reports. 

Internal  Control.  Will  require  OSD 
to  ensure  that  prototyping  plans  are 
included  in  the  cost  and  operational 
efficiency  analysis  and  life-cycle 
cost  estimates. 


Amount  and/or 
Type  of  Benefit 


Nonmonetary. 


Nonmonetary. 


Nonmonetary. 


Nonmonetary. 
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Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology,  Washington,  DC 
Comptroller  of  the  Department  of  Defense,  Washington,  DC 

Assistant  Secretary  of  Defense  (Command,  Control,  Communications  and  Intelligence), 
Washington,  DC 
Director,  Defense  Procurement 

Director,  Program  Analysis  and  Evaluation,  Washington,  DC 
Ballistic  Missile  Defense  Organization,  Washington,  DC 


Department  of  the  Army 

Assistant  Secretary  of  the  Army  (Research,  Development  and  Acquisition), 
Washington,  DC 

U.S.  Army  Operational  Test  and  Evaluation  Command,  Alexandria,  VA 
U.S.  Army  Program  Executive  Office  Tactical  Systems,  Redstone  Arsenal,  AL 
U.S.  Army  Program  Office,  Brilliant  Anti-Armor  Submunition,  Redstone  Arsenal,  AL 
U.S.  Army  Program  Office,  Comanche  Helicopter,  Saint  Louis,  MO 
U.S.  Army  Program  Office,  Combat  Service  Support  Control  System,  Fort  Belvoir, 
VA 

U.S.  Army  Program  Office,  Kiowa  Warrior,  Saint  Louis,  MO 

U.S.  Army  Program  Office,  Palletized  Loading  System,  Warren,  MI 

U.S.  Army  Program  Office,  Secure  Mobile  Anti- Jam  Reliable  Tactical  Terminal, 

Fort  Monmoth,  NJ 

U.  S.  Army  Program  Office,  Short  Range  Unmanned  Aerial  Vehicle,  Huntsville,  AL 
U.S.  Army  Program  Office,  Single  Channel  Anti -Jam  Manportable  Terminal,  Fort 
Monmoth,  NJ 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition), 
Washington,  DC 

Naval  Operational  Test  and  Evaluation  Force,  Norfolk,  VA 
F/A-18  E/F  Program  Office,  Naval  Air  Systems  Command,  Washington,  DC 
Fixed  Distributed  System  Program  Office,  Space  and  Naval  Warfare  Systems 
Command,  Washington,  DC 

Joint  Stand-Off  Weapon  Program  Office,  Naval  Air  Systems  Command, 
Washington,  DC 
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Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Acquisition),  Washington,  DC 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Controller), 
Washington,  DC 

Air  Force  Operational  Test  and  Evaluation  Center,  Albuquerque,  NM 

System  Program  Office,  E-3  Airborne  Warning  and  Control  System  -  Radar  System 
Improvement  Program,  Hanscom  Air  Force  Base,  MA 

System  Program  Office,  F-22  Advanced  Tactical  Fighter,  Wright-Patterson  Air  Force 
Base,  OH 

System  Program  Office,  Joint  Direct  Attack  Munitions  System,  Eglin  Air  Force 
B3,sc  FL 

System  Program  Office,  Joint  Primary  Aircraft  Training  System,  Wright-Patterson  Air 
Force  Base,  OH 

System  Program  Office,  Joint  Service  Imagery  Processing  System,  Hanscom  Air  Force 
Base,  MA 


Other  Defense  Organizations 

Defense  Advanced  Research  Projects  Agency,  Arlington,  VA 
Defense  Logistics  Agency,  Cameron  Station,  Alexandria,  VA 
Defense  Plant  Representative  Office  -  The  Boeing  Company,  Seattle,  WA 
Defense  Plant  Representative  Office  -  Lockheed  Aeronautical  Systems  Company, 
Marietta,  GA 

Defense  Plant  Representative  Office  -  McDonnell  Douglas  Aerospace  East,  Saint 
Louis,  MO 

Defense  Plant  Representative  Office  -  McDonnell  Douglas  Helicopter  Systems,  Mesa, 
AZ 

Defense  Plant  Representative  Office  -  Northrop  Corporation,  Pico  Rivera,  CA 
Defense  Plant  Representative  Office  -  Sikorsky  Aircraft,  Trumbull,  CT 
Defense  Plant  Representative  Office  -  Texas  Instruments,  Dallas,  TX 


Non-Defense  Organizations 

The  Boeing  Company,  Seattle,  WA 
Institute  for  Defense  Analysis,  Alexandria,  VA 

Lockheed  Corporation,  L<xkheed  Aeronautical  Systems  Company,  Marietta,  GA 

McDonnell  Douglas  Aerospace  East,  Saint  Louis,  MO 

McDonnell  Douglas  Helicopter  Systems,  Mesa,  AZ 

Northrop  Corporation,  Northrop  Aircraft  Division,  Pico  Rivera,  CA 

RAND  Corporation,  Santa  Monica,  CA 

Texas  Instruments  Incorporated,  Dallas,  TX 

United  Technologies  Corporation,  Sikorsky  Aircraft  Division,  Trumbull,  CT 
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Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Deputy  Under  Secretary  of  Defense  for  Acquisition  Reform 
Director,  Defense  Procurement 
Comptroller  of  the  Department  of  Defense 

Assistant  Secretary  of  Defense  (Command,  Control,  Communications  and  Intelligence), 
Washington,  DC 

Assistant  to  the  Secretary  of  Defense  for  Public  Affairs 
Director,  Ballistic  Missile  Defense  Organization 


Department  of  the  Army 

Secretary  of  the  Army 

Assistant  Secretary  of  the  Army  (Research,  Development  and  Acquisition) 
Director,  U.S.  Army  Oi)erational  Test  and  Evaluation  Command 
Auditor  General,  Department  of  the  Army 
U.S.  Army  Program  Executive  Officer,  Tactical  Systems 

U.S.  Army  Project  Manager,  Brilliant  Anti-Armor  Submunition 
U.S.  Army  Project  Manager,  Comanche  Helicopter 


Department  of  the  Navy 

Secretary  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management) 

Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition) 
Comptroller  of  the  Navy 

Commander,  Naval  Operational  Test  and  Evaluation  Force 
Auditor  General,  Department  of  the  Navy 
Program  Manager,  F/A-18  E/F,  Naval  Air  Systems  Command 
Program  Manager,  Joint  Stand-Off  Weapon  System,  Naval  Air  Systems  Command 
Program  Manager,  Fixed  Distributed  System,  Space  and  Naval  Warfare  Systems 
Command 
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Department  of  the  Air  Force 

Secretary  of  the  Air  Force 

Assistant  Secreta^  of  the  Air  Force  (Acquisition) 

Director,  Audit  Liaison  and  Follow-Up 
Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Air  Force 
Commander,  Air  Force  Operational  Test  and  Evaluation  Center 
Program  Manager,  F-22  Advanced  Tactical  Fighter 


Other  Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 
Director,  Advanced  Research  Projects  Agency 
Director,  Defense  Logistics  Agency 

Commander,  Defense  Plant  Representative  Office  -  The  Boeing  Company 
Commander,  Defense  Plant  Representative  Office  -  Lockheed  Aeronautical  Systems 
Company 

Commander,  Defense  Plant  Representative  Office  -  McDonnell  Douglas  Aerospace 
East 

Commander,  Defense  Plant  Representative  Office  -  McDonnell  Douglas  Helicopter 
Systems 

Commander,  Defense  Plant  Representative  Office  -  Northrop  Corporation 

Commander,  Defense  Plant  Representative  Office  -  Sikorsky  Aircraft 

Commander,  Defense  Plant  Representative  Office  -  Texas  Instruments 

Director,  National  Security  Agency 

Inspector  General,  Central  Imagery  Office 

Inspector  General,  Defense  Intelligence  Agency 

Inspector  General,  National  Security  Agency 

Director,  Defense  Logistics  Studies  Information  Exchange 


Non-Defense  Federal  Organizations 

Office  of  Management  and  Budget 

U.S.  General  Accounting  Office,  National  Security  and  International  Affairs  Division, 
Technical  Information  Center 

Chairman  and  Ranking  Minority  Member  of  Each  of  the  Following  Congressional 
Committees  and  Subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
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Non-Defense  Federal  Organizations  (Continued) 

House  Committee  on  Government  Operations 

House  Subcommittee  on  Legislation  and  National  Security,  Committee  on 
Government  Operations 
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Part  IV  -  Management  Comments 


Under  Secretary  of  Defense  for  Acquistion  and 
Technology  Comments 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

30CX>  OEFENS^E  PENTAGON 
WASHINGTON  DC  *0301-3000 

f  3  M  |0p4 

’nCCHNOCOOY 

A*X/XSM 


MEMORANDUM  FOR  DIRECTOR,  ACQUISITION  MANAGEMENT  DIRECTORATE, 
DoD  INSPECTOR  GENERAL 

SUBJECT:  The  Effectiveness  of  Prototyping  Acquisition 

Strategies  for  Major  Defense  Acquisition  Prograa 
(Project  No.  2AE-0051) 


I  have  reviewed  your  draft  summary  report  dated  March  30, 
1994,  and  offer  the  following  overall  response  to  your  findings 
and  recommendations.  The  findings  are  extremely  weak  and  I  non¬ 
concur  with  all  13  recommendations  except  for  two  that  I 
partially  concur  in  and  only  one  other  that  I  concur  with.  In 
general,  I  am  disappointed  in  the  lack  of  constructive 
recommendations  from  this  extensive  effort. 

I  agree  that  prototyping  could  be  used  more  effectively 
reduce  risk  and  assess  how  the  design  approach  will  operate 
before  proceeding  with  EMD.  However,  I  strongly  disagree  with 
your  proposal  to  radically  reduce  the  flexibility  currently 
provided  to  the  Milestone  Decision  Authority  (MDA)  to  choose  an 
acquisition  strategy  appropriate  to  the  specific  situation.  You 
recommend  very  significant  5000.2  changes  (Milestone  1/11/  IV, 
early  operational  test,  acquisition  baseline,  exit  criteria,  TEMP 
software  plan,  prototyping  plan,  DoDS  2167A,  producibility 
analysis,  and  the  creation  of  a  new  handbook).  You  also 
recommend  revision  to  the  FAR,  and  a  greatly  increased  role  for 
ARPA  and  PAfcE  in  the  DAB  process  based  on  weak  and  unconvincing 
rationale.  These  recommendations  would  add  significantly  and 
unnecessarily  to  bureaucratic  oversight,  and  appear  to  conflict 
with  Secretary  Perry’s  recent  memo  regarding  waiver  authority, 
streamlining,  and  reduction  of  bureaucratic  regulations.  Because 
additional  prototyping  requires  additional  up-front  money  and 
acquisition  strategics  must  be  carefully  constructed,  your 
proposals  could  ultimately  reduce  the  capability  of  the  Xraed 
Forces  by  diverting  funds,  for  example,  from  production  to  low- 
payoff  mandatory  prototypes. 

I  would  like  to  call  your  attention  to  how  little  new  data 
you  have  in  your  report.  There  are  only  approximately  10  pages 
out  of  your  63  page  report  that  purports  to  give  any  new  insight 
into  prototyping.  An  inordinate  amount  of  the  report  simply 
states  what  is  written  in  5000.2,  what  RAND  and  IDA  concluded, 

DMR  recommendations,  and  Packard  Report  recommendations.  We  are 
quite  familiar  with  these. 

O 
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I  an  disappointed  by  the  evaluations  that  you  conducted  on 
the  6  MDAPs  (UTTMDS,  Comanche,  BAT,  JSOW,  FDS,  and  P-22).  There 
are  many  inaccuracies  in  your  findings  on  UTTMDS,  JSOW,  and  FDS, 
as  well  as  essentially  no  data  on  F-22  and  Comanche.  In 
particular  your  report  is  unbalanced  to  the  ejftent  that  it  only 
illuminates  problems  with  the  aforementioned  MDAP  prototypes,  and 
largely  ignores  the  very  large  (and  e)cpensive)  and  very 
successful  F-22  Dem/Val  competitive  prototyping  effort  that 
included  an  actual  ••fly-off"  between  the  prototypes.  I  also  note 
there  is  a  dearth  of  information  on  Comanche.  Given  the  status 
of  Comanche  as  one  of  the  few  remaining  Army  major  acquisition 
programs,  I  feel  that  you  should  have  devoted  substantial 
attention  to  its  prototyping  approach. 

In  summary,  I  have  considerable  concerns  about  your 
misunderstanding  of  the  need  for  the  flexibility  inherent  in  the 
current  5000  series  directives  and  the  DoD  acquisition  process  in 
general  as  noted  in  the  attached  detailed  comments.  I  suggest 
you  consider  withdrawing  this  draft  report. 


Director,  Acquisition 
Program  Integration 

Attachment 
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Final  Report 
Reference 


10 


12 


12-13 


Attachment 


•  On  page  12,  the  section  on  Hardware  indicates  that  four  of  the  six 
programs  reviewed,  including  the  Army  BAT  and  Comanche  programs, 
and  the  Navy  FDS  and  JSOW  programs,  had  limitations  in  planning  or 
execution  of  hardware  prototyping  strategies.  However,  there  is  no 
discussion  in  this  section  on  what,  if  any,  limitations  FDS  had  in 
planning  or  execution  of  hardware  prototyping  strategies. 

•  On  page  14,  the  report  indicates  inaccurately  that  the  JSOW  program 
did  not  plan  to  use  prototyping  in  Dea/Val.  The  JSOW  program 
planned  on  an  appropriate  amount  of  component  prototyping  using  the 
Recoverable  Test  Vehicle  Program.  When  that  program  ran  into 
difficulties,  the  JSOW  program  adapted,  and  coordinated  with  all 
three  Dem/Val  contractors  to  modify  their  plan  to  include  missile 
prototypes.  These  contractors,  using  their  own  funds,  invested  in 
their  own  development  programs  to  demonstrate  the  maturity  of  their 
systems,  recognizing  the  competitive  environment  and  considering 
the  potential  for  future  contracts.  Investing  IRtD  funds  and  other 
corporate  resources  is  a  normal  commercial  activity.  This  occurred 
at  a  time  when  DoD  was  encouraging  contractors  to  participate  in 
the  risks  of  development.  As  a  result,  the  cost  to  the  government 
of  the  three,  fixed  price  Dem/Val  contracts  was  $1.5  million.  Had 
the  JSOW  program  contracted  for  three  independent  prototyping 
efforts  as  proposed  by  the  DoDIG  report,  the  impact  would  have  been 
an  additional  $64  million  and  a  delay  of  2  years. 

Although  the  report  recognizes  prototyping  is  primarily  a  risk 
reduction  activity,  the  report  fails  to  recognize,  primarily  on 
pages  14  and  15,  that  the  whole  premise  of  the  JSOW  program  is  low 
risk  integration  of  existing  technology.  Experienced  program 
managers  and  engineers  did  assess  program  risks  and  developed  the 
prototyping  strategy  accordingly.  This  strategy  was  scrutinized  by 
ASN  and  OSD,  determined  to  be  sound,  and  hindsight  only 
demonstrates  the  validity  of  those  actions.  The  JSOW  program  is 
now  in  EMD,  and  the  use  of  prototypes  is  continuing,  with 
Structural  Test  Vehicles,  Advanced  Development  Models,  Engineering 
Development  Models,  and  Production  Representative  Models  all  being 
produced  to  support  the  necessary  testing  at  the  appropriate  time. 

-  All“up-round  prototyping  was  not  part  of  the  JSOW  Dem/Val 
effort  because  the  Baseline  JSOW  vehicle  was  considered  low  risk 
as  other  similar  vehicles  were  being  tested  in  the  Navy  and  Air 
Force.  Based  on  the  estimated  cost  of  further  prototype  testing 
for  three  Dem/Val  competitors,  it  was  not  considered  cost 
effective  to  do  this  prototyping  for  the  additional  information 
which  would  be  gained  , 

•  It  was  determined  at  Milestone  I  that  conducting  subsystem 
testing,  analysis,  simulations,  and  demonstrations  in  Dem/Val 
would  be  adecfuate  to  support  the  risk  assessment  and  a  decision  to 
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Final  Report 
Reference 


move  into  Engineering  and  Manufacturing  Developxaent,  These 
included: 

—  Wind  tunnel  tests  to  validate  kinematics  range 

—  Launch  separation  analysis  and  wind  tunnel  tests  to 
validate  separation  effects 

—  3-DOF/6-DOr  simulations  for  guidance,  control,  and  range 
information 

—  Rardvare-in*-the-loop  testing 

—  Ground  testing  of  radar  cross  section 

--  Analysis  and  ground  testing  of  payload  dispenser 
mechanization 

—  Targeting,  accuracy  and  lethality  analysis 

--  Producibility  analysis/studies,  structures,  materials, 
loads  and  handling  tests,  and  airframe/ving  deployment 
tests 

-  The  three  competing  contractors  were  given  the  option  to 
conduct  captive  and  free  flight  tests  for  demonstration  if  they 
desired.  Each  did  conduct  a  free  flight  demonstration  of  their 
vehicle  design. 

•  On  page  16,  •'Shore  Signal  Information  Processing  System"  should 
read,  "Shore  Signal  Information  Processing  Segment." 

•  On  page  16,  the  report  states,  "Prototyping  software  was  not 
developed  and  tested  during  Phase  I,  Dem/Val  for  the  underwater 
segment  of  the  FDS."  Prototyping  of  software  was  not  performed 
because  there  are  only  2500  lines  of  code  in  the  underwater 
segment  of  FDS.  The  level  of  risk  did  not  warrant  the  added  cost 
of  prototyping  software. 

•  On  page  18,  the  report  does  not  recognize  that  while  some 
manufacturing  technologies  were  developed  for  the  FDS  program, 
formal  "prototyping"  was  not  accomplished  because  insufficient 
funds  were  available.  Furthermore,  the  FDS  program  office 
believes  that  manufacturing  technology  was  in  accordance  with  DoD 
Directive  4245.6  during  demonstration  validation  of  FDS. 

•  On  page  19,  the  report  states  that  the  JSOW  program  failed  to 
comply  with  Title  10,  United  States  Code,  Section  2365,  which 
requires  a  waiver  request  if  prototyping  is  not  to  be  used.  The 
JSOW  program  office  maintains  it  was  conducting  an  appropriate 
amount  of  prototyping.  Furthermore,  the  issue  was  raised  at  the 
Milestone  I  DAB  on  June  5,  1989,  and  specifically  addressed  in  the 
Executive  Summary  of  the  decision  memorandum.  USD (A)  stated  that 
the  prototyping  strategy  complied  with  DoD  policy  and  approved 
the  Milestone  I. 


14 


14 


16 


16 
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17-18 


35 


50 


20, 

29, 

61 


•  Your  finding  on  conpetitiv*  prototyping  by  iubcontr actors, 
addressed  on  pages  19  through  21,  deals  with  conpetitivs 
prototyping,  focused  on  key  subsysteas,  restricted  to  the 
subcontractor  level.  You  contend  that  "the  winning  subcontractor 
aust  be  selected  based  on  the  best  interest  of  the  goverr^ant," 
and  comoend  the  Xrioy  BAT  prograa's  approach  to  "control [ling]  the 
selection  of  the  winning  subcontractors."  Specifically,  "(t]o 
saintain  control  over  subcontractor  coapetition,  the  BAT  Pro^aa 
Office  .  .  .  inserted  provisions  into  the  Request  for  Proposals  . 

.  .  that  allow  the  governaent  to  establish  the  selection  criteria 
to  be  used  for  choosing  the  subcontractors  .  ,  .  [and)  provide 
that  no  author  i  rat  ion  can  be  given  to  proceed  with  a  sxibcontract 
until  the  government  has  approved."  You  view  this  as  a  nacessary 
element  of  "effective  management  of  subcontractors"  and  state  that 
"similar  contractual  provisions  should  be  used  in  all  acquisition 
programs  where  competitive  prototyping  is  limited  to  the 
subcontractor  level.*'  This  discussion  is  not  translated  into  an 
IG  Recommendation. 

There  are  numerous  situations  in  which  the  government  has  awarded 
a  prime  contract,  and  has  a  critical  interest  in  the  prime's  post- 
award  selection  of  a  subcontractor  for  a  particular  subsystem  or 
component.  In  these  cases,  vc  agree  that  the  prime  contract 
should  require  government  approval  of  the  prime's  source  selection 
plan,  including  selection  criteria,  before  the  prime  Issues  its 
solicitation.  Through  normal  contract  administration,  the 
government  can  ensure  the  prime  complies  with  the  epproved  plan. 

We  disagree  that  the  government  should  "control"  the  subcontract 
competition,  or  subject  the  prime's  selection  to  government 
approval  or  veto.  This  is  inconsistent  with  holding  the  prime 
responsible  for  performance  of  the  total,  integrated  system.  If 
this  degree  of  control  is  required,  the  subsystem  or  component 
should  be  provided  as  government  furnished  equipment  after  a 
government-conducted  source  selection. 

•  On  page  37,  Recommendation  l.a.(3)  should  be  rewritten  to  limit 
ARRA  review  to  the  prototyping  plans  rather  than  to  the  entire 
Acquisition  Strategy  Reports, 

•  On  page  42,  footnote  2,  regarding  JPATS,  is  incorrect.  There  is 
no  JPATS  contract.  The  Air  Force  has  recently  issued  the  Request 
for  Proposals  intended  to  lead  to  the  first  JPATS  contract. 

•  The  statements  on  pages  22,  23,  32,  and  54  that  the  DTTKDS  APB  did 
not  include  dates  of  delivery  of  prototype  missiles  are  incorrect. 
The  40  User  Operational  tvaluetion  System  (UOES)  misalle  delivery 
dates  arc  in  the  UTTMDS  APB,  and  the  THAAD  Provisional  Performance 
Criteria  (PPC)  has  UOES  test  start  dates  as  well  as  delivery 
dates.  The  PPC  is  equivalent  to  an  APB. 
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•  I  disagree  with  your  conclusion  on  page  24  that  a  definition  of 
prototyping  is  needed  in  5000.2,  with  your  only  rationale  being 
that  it  is  in  a  RAND  report.  Which  of  the  six  KDXPs  you  evaluated 
suffered  from  lack  of  a  definition?  I  feel  prototyping  is  a  veil 
known  concept. 


•  What  new  IC  data  supports  your  statement  on  page  29  that 
deficiencies  observed  in  MDAP  strategies  were  due  to  lack  of  ARPA 
involvement?  Your  discussion  only  references  the  ISM  Packard 
Report  and  offers  no  other  substantiation  of  your  finding.  ARPA 
does  not  have  the  knowledge  or  expertise  in  HDAPs  that  is  resident 
in  the  Services'  and  BMDO  organizations.  While  ARPA's  role  in  the 
technology  base  area  is  important,  the  services  and  BMDO  correctly 
argue  that  ARPA  is  not  resourced  or  staffed  to  be  the  single  DoD 
focal  point  for  jlH  prototyping. 

•  I  take  issue  with  your  characterization  of  existing  guidance  on 
page  26. 

DoDI  5000.2  shows  a  clear  preference  for  prototyping  and  a 
requirement  to  discuss  prototyping  in  addressing  the  recommended 
acquisition  approach. 

-  Paragraph  3.f.(l)  of  Section  5-C  states,  "Protot^ing  will  be  a 
major  element  of  Phase  1,  Demonstration  and  Validation."  Note 
that  this  sentence  is  not  limited  to  competitive  prototyping. 
Paragraph  3.f  continues  with  a  discussion  of  how  to  do  prototyping 
and  the  focus  of  prototyping. 

-  The  Acquisition  Strategy  Report  in  Section  4-D  of  DoD  5000. 2-M 
contains  a  requirement  for  discussing  the  accpiisition  strategy 
including  the  requirement  to  prototype.  That  strategy  must  be 
approved  by  the  milestone  decision  authority.  A  decision  not  to 
use  prototyping  must  be  justified  in  the  ASR. 

-  Part  12  of  DoD  5000. 2-M  was  removed  from  the  manual  because  it 
contained  a  format  for  submitting  waivers  of  competitive 
prototyping  to  Congress.  Congress  removed  the  requirement  for 
waiver  notification  when  it  reinstated  the  CPS  requirement. 

•  I  find  essentially  no  discussion  of  Comanche  in  the  findings,  even 
though  you  say  on  page  5  that  you  evaluated  it.  What  did  you 
find?  Comanche  plans  5  "development  aircraft":  2  EMD  prototypes; 
and  3  LRIP  aircraft.  This  is  the  so-called  "streamlined"  program; 
the  original  program  called  for  3  Dem/Val  prototypes  and  3  EKD 
prototypes.  Army  claims  it  can  cut  out  the  number  of  prototypes 
because  of  increased  reliance  on  modeling  and  simulation. 

Several  OSD  officials  are  skeptical  about  this  approach.  The  2 
EMD  prototypes  are  scheduled  as  follows:  AV-01  has  first  flight 
in  early  FYSe;  and  AV-02  has  first  flight  in  a  little  over  two 
years.  But  that's  it  until  the  first  LRIP  test  aircraft  which 
doesn't  have  first  flight  until  2002.  Also,  this  strategy  pushes 
a  great  deal  of  risk  off  on  the  LRIP  decision  because  Army  isn't 
doing  final  development  testing  until  the  UlIP  helicopters. 
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R»comgfB(a»tion  IfMfl 

I  non-concur  with  the  recommendation  to  add  definition  of 
prototyping  to  5000.2.  Prototyping  is  a  well  Known  concept. 


PeconatBdatiop 

I  non-concur  with  the  recommendation  to  revise  5000.2  to 
require  the  PM  to  submit  a  prototyping  plan  for  Milestone  I.  The 
plan  is  an  inherent  part  of  the  ASR  and  TEMP  already.  Additional 
oversight  is  not  justified  by  your  findings,  and  runs  counter  to 
the  streamlining  required  by  Executive  Order  12861.  Adding  to  the 
PM  workload  is  also  contrary  to  KPR  efforts. 

No  quarrel  can  be  made  for  the  need  for  prototyping  (though 
traditionally  satellite  programs  and  Navy  shipbuilding  programs  do 
not  use  prototyping)  .  However,  there  remains  the  question  of  how 
prototyping  should  be  done. 

At  present,  there  is  a  significant  effort  by  the  Acquisition 
Task  Force  on  Modeling  and  Simulation  (ATFMS)  to  use  computer¬ 
generated  models  in  lieu  of  actual  soft-tooled  prototypes.  This 
approach  could  greatly  shorten  the  acquisition  cycle  and  could 
provide  a  much  less  costly  approach  than  building  actual 
prototypes.  Before  accepting  your  far-reaching  recommendations,  we 
should  wait  until  the  ATFMS  has  had  an  opportunity  to  present  its 
findings  to  the  OSD  leadership.  That  effort  should  be  completed 
within  the  next  90  days. 


Recommendation  I.a,3 

I  non-concur  with  the  recommendation  to  revise  5000.2  to  have 
ARPA  report  to  MDA  on  ^11  ACAT  Is  and  IIs  before  Milestone  I.  I 
see  no  strong  rationale  supporting  this  for  MDAPs  or  ACAT  IIs,  as 
noted  in  our  comments  on  your  page  29  finding.  The  AJIPA  role  is 
technology  based. 


Recommendation 

I  non-concur  with  the  recommendation  to  revise  5000.2  to  have 
ARPA  report  to  MDA  on  ill  ACAT  Is  and  IIs  before  Milestone  IV  for 
the  reason  given  in  l.a.3  above.  Producibility  will  be  very 
important  at  Milestone  IV,  and  I  epestion  whether  ARPA  is  the 
correct  organization  to  review  this. 


Recommendation  X.a.S 

I  partially  concur  with  the  recommendation  to  revise  5000.2  to 
require  Directors  of  Military  Departments*  OT6E  organizations  to 
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report  to  MDA  on  prototyping  requirements.  D,OT4E  sees  benefit  in 
this,  but  Army  and  Navy  have  some  cjuestions  as  to  how  this  would  be 
accomplished . 


Recoamepditiop  l>>,g 

I  concur  with  your  recommendation  to  revise  5000.2  to  add 
prototype  schedule  dates  to  the  DAES  baseline « 


Racommsadation  l.ir? 

I  non-concur  with  your  recommendation  to  revise  5000.2  to 
require  exit  criteria  for  prototyping  during  phase  I,  Requiring 
specific  exit  criteria,  design  reviews,  and  producibility  analysis 
gives  no  leeway  to  the  program  manager  or  the  Milestone  Decision 
Authority  (MDA) .  One  of  the  basic  tenets  of  the  revised  DoD  5000 
series  was  to  provide  sufficient  guidance  and  then  allow  leeway  to 
the  program  manager  and  the  MDA  to  run  the  progrram  in  a  manner  that 
reflects  good  business  practice.  Requiring  certain  steps ^ 
regardless  of  the  program  particulars  appears  to  be  a  radical 
change  from  this  basic  tenet.  Requirements  should  be  couched  in  a 
manner  that  reflects  that  these  actions  are  necessary  to  support 
the  use  of  prototypes,  rather  than  having  a  blind  requirement  for 
each  program. 

This  recommendation  presupposes  that  all  programs  are 
prototyped,  which  is  not  li)cely. 


Recommendation  l.a.S 

I  non-concur  with  your  recommendation  to  revise  5000,2  to 
require  that  an  initial  review  based  on  prototyping  be  completed 
before  Milestone  II.  My  non-concurrence  is  similar  to  l.a.7  above 
.  .  .  not  all  programs  are  prototyped. 

The  existing  decision  criteria  specified  in  DoDI  5000,2  for 
Milestone  II  make  this  requirement  unnecessary.  The  decision 
authority  must  confirm  that  "Prototyping  and  Demonstration  results 
to  date  provide  reasonable  assurance  that  the  technologies  and 
processes  critical  to  success  are  attainable." 


RecommesdatioD 

I  non-concur  in  your  recommendation  to  revise  5000.2  to 
require  a  producibility  analysis  be  prepared  based  on  prototypes. 
Again,  not  all  programs  will  be  prototyped. 

A  producibility  analysis  is  normally  prepared  during 
Engineering  and  Manufacturing  Development  to  support  the  Production 
Readiness  Reviews.  Such  analysis  is  normally  based  on  the 
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engineering  development  model,  but  the  prototype  can  also  be  used 
for  input  into  the  analysis. 


Recoaaepdatiop 

1  non-concur  in  your  recommendation  to  revise  5000.2  to 
require  a  software  TfcE  plan  to  be  part  of  the  TEMP  for  all 
milestone  decisions. 

Software  test  and  evaluation  is  currently  required  to  be  an 
integral  part  of  the  Test  and  Evaluation  Master  Plan.  What  do  you 
propose  to  revise? 


peconmendatioa  l.a.ll 

I  partially  concur  in  your  recommendation  to  revise  5000.2  to 
require  RFPs  for  Dem/Val  contracts  include  tailoring  of  DoD 
Standard  2167A.  It  is  inappropriate  to  require  the  use  of  a  DoD 
Standard  when  commercial  standards  are  preferable  according  to  Dep 
Sec  Def  guidance. 

PtcomneDdation 

I  non-concur  in  your  recommendation  to  revise  5000.2  to  issue 
a  guidance  handbook  on  prototyping.  While  it  is  not  clear  that  a 
handbook  is  necessary  ,  it  is  certainly  not  necessary  to  revise 
5000.2  to  accomplish  this. 

Recommepdatlop  l.c 

I  non-concur  in  your  recommendation  to  reinstate  guidance 
deleted  in  the  February  1993  Change  1  to  5000. 2-M. 

Change  1  to  DoD  5000. 2-M  incorporated  the  requirement  to 
include  the  rationale  in  the  Acquisition  Strategy  Report,  when  it 
is  not  practicable  to  use  a  competitive  prototyping  strategy.  See 
our  comments  on  your  page  26  findings. 


Recommendatiop  2 

We  non-concur  in  your  recommendation  that  PA&E  report  on 
sufficiency  of  the  proposed  prototyping  plan  at  Milestone  I.  I 
non-concurred  in  the  requirement  for  a  formal,  stand-alone 
prototyping  plan,  so  there  is  no  need  for  this.  I  understand  that 
PAtE  is  also  non-concurring  under  separate  cover  to  you. 


Under  Secretary  of  Defense  for  Acquisition  and  Technology  Comments 


R»contt»pd»tiQli^ 

I  partially  concur  in  your  recommendation  to  revise  the  DFAK5 
to  provide  solicitation  language  and  DIDs  development  to  document 
results  of  prototyping. 

Regarding  the  first  point,  the  Director,  Defense  Procnir^ent 
suggests  that  your  purpose  would  be  better  served  by  Indicating  in 
the  suggested  DoDI  5000.2  language  on  prototyping  that  the 
solicitation's  proposal  preparation  instructions  Include  a  complete 
description  of  the  prototyping  plan  required  to  be  submitted  with 
the  proposal.  This  type  of  solicitation  requirement  will  have  to 
be  tailored  for  each  program,  and  is  not  susceptible  to 
standardization.  Xny  "standard"  DFARS  provision  would  become  a 
redundant  "boilerplate,"  while  the  real  information  about  what  is 
required  would  appear  in  proposal  preparation  instjructions  anyway. 
Consequently,  we  also  recommend  that  the  following  sentence  be 
deleted  from  "Contractor  Input  to  Prototyping  Plans"  on  page  29; 

"To  accomplish  those  objectives  the  [DFARS]  should  be  amended  to 
provide  the  solicitation  language  to  use  in  the  Request  for 
Proposals. " 

DDP  also  reports  that  they  do  not  have  a  role  in  the 
authorization  or  preparation  of  data  item  descriptions  (DIDs) .  In 
an  effort  to  identify  an  appropriate  office  to  which  this 
recommendation  could  be  addressed,  they  learned  that  there  is  no 
obvious  addressee.  A  DID  is  originated  by  the  organization  needing 
data  delivered  under  a  contract  that  cannot  be  obtained  through  use 
of  an  existing  DID  listed  in  the  "Acquisition  Management  Systems 
and  Data  Requirements  Control  List,"  DoD  5010, 12-L.  If  USD(AtT) 
agrees  that  DIDs  should  be  used  as  the  IG  recommends,  then  he 
should  designate  an  Office  of  Primary  Responsibility  in  one  of  the 
Components  and  direct  it  to  verify  no  existing  DIDs  would  suffice 
before  developing  new  ones. 
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OFFICE  OF  THE  SECRETARY  OF  DEFENSE 
1  BOO  DEFENSE  PENTAGON 
WASHINGTON  DC  2030 1 -I  BOO 


03  JUN  T994 


MEMORANDUM  FOR  THE  INSPECTOR  GENERAL  OF  THE  DEPARTMENT  OF 
DEFENSE 

SUBJECT:  Draft  Inspector  General  (IG)  Audit  Report  on  The  Effectivcnc*i  of  Prototyping 

Acquisition  Strategies  for  Major  Defense  Acquis  don  Prognunj  (MDAP),  (Project 
No  2AE-0051) 

Thank  you  for  the  opportunity  to  comment  on  the  finding  and  recommendations  in  the 
draft  repon  on  prototyping.  1  cannot  concur  with  the  cuircni  verson  of  the  report:  it 
contains  no  clearly  identifiable  statement  of  findings;  and,  to  the  extent  that  a  finding  can  be 
infcTTed  the  repon  lacks  compelling  supporting  rationale.  The  remainder  of  thia 
memorandum  discusses  issues  that  should  be  clarified  in  a  reviaed  draft. 

Inferred  Finding  and  the  RAND  Prototyping  Study.  The  "CoDclufions**  section  of  the 
draft  states:  "Effective  prototyping  to  reduce  program  risk  in  trananoning  from  DenVVal  to 
EMD  can  only  occur  through  the  development  and  implementation  of  a  structured 
decisionmaking  process  that  will  result  in  a  documented  and  executable  prototyping  plan.* 
Assuming  that  this  is  a  valid  siaiemeni  of  the  draft  report's  finding,  it  appears  to  be  supported 
primarily  by  reference  to  the  RAND  report  (R-4161-ACQ)  The  Nature  and  Role  of 
Prototyping  in  WeaTX)n  System  PevelopmenL  The  following  citarions  taken  from  the  RAND 
study  seem  to  argue  against,  rather  than  for,  highly  structured  prototype  planning,  however 

On  deciding  via  policy  whether  or  not  to  prototype,  the  RAND  repon  sqyj.  This 
analysis  suggests  that  attempts  to  define  such  a  policy  ax  any  but  the  broadest  levels  should  be 
avoided  ...  Thus,  it  is  not  possible  or  even  desirable  to  develop  a  set  of  firm  decision  raid." 

On  performing  the  cost  versus  benefits  anoiysis  of  prototyping  that  the  draft  repon 
recommends  be  mandated  in  DODl  50002,  the  RAND  repon  saytt  "In  summary,  there  are 
tremendous  difficuldes  involved  in  operationalizing  the  simple  concept  depicted  in  Figure  6.1 
(cost  versus  benefit).  We  cannot  be  confident  that  we  can  identify  all  possible  benefits  and 
costs  ...  Even  if  we  could  measure  all  relevant  costs  and  benefits,  we  do  not  know  how  to 
consistenily  weigh  them  in  a  decision  process.* 

On  trying  to  analyze  costs  versus  benefits  of  prototyping,  the  RAND  repon  says,  "In 
the  end,  there  is  no  substitute  for  informed  judgment  made  by  experienced  managers  and 
engineers.” 

Regarding  current  (1992)  DoD  regulations,  the  RAND  repon  says.  This  conceptual 
framework  for  policy  and  decisioa  making  is  quite  similar  to  what  is  contained  in  existing 
regulations...";  and,  "From  the  perspective  of  this  research,  the  expressed  policy  of  DoD  is 
consd stent  with  the  conceptual  framework  developed  both  in  this  section  and  in  Section  2, 
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The  base  implication,  then,  is  that  acquisition  policy  should  reflect  only  broad  gaidelinei  on 
prototyping,  ” 

On  prototyping  versus  non^proioryping,  the  RAND  report  says,  "However,  there  ire 
few  significani  differences  between  prototyping  and  non-prototyping  programs  with  ie^)ect  to 
cost  growth,  total  actual  program  duration,  or  icbedule  slip." 

A  revised  repon  should  reconcile  the*  staicmenu  with  the  references  to  the  Rand  study 
appearing  in  the  current  draft  asserting  that  the  present  acquisition  regulatioci  regarding 
prototyping  are  inadequate,  and  that  a  highly  structured  process,  incorporating  a  cost-benefit 
analysis,  should  be  adopted. 

The  IDA  Froiorypirtg  Study.  The  draft  repon  also  presents  data  taken  from  an  IDA 
study  (actually,  a  scripted  briefing)  indicating  that  development  and  production  cost  growth  is 
significantly  less  for  programs  using  prototypes  than  for  programs  excluding  prototypes.  The 
IDA  repon  does  not  assen  this  conclusion  unequivocally,  however.  It  notes  that  prototyping 
is  significantly  beneficial  for  munitions  programs,  with  the  benefits  for  aiicrafl  programs 
being  less  apparent-  Moreover,  many  of  the  programs  counted  by  IDA  u  excluding 
prototypes  were  modifications  to  existing  systems;  but,  the  existing  systems  could  arguably  be 
considered  prototypes  of  the  subsequent  modifications.  Given  the*  caveats  and  ti* 
conclusions  of  the  RAND  study,  the  evidence  supporting  the  draft  report’s  finding  and 
recommendations  appean  ambivalent. 

The  Draft  Report's  Review.  The  draft  report  lists  six  programs  as  subjects  of  a  review 
by  the  IG  staff,  but  discusses  only  five;  three  of  the  five  programs  are  disetmed  in  detaiL 
Two  of  the  fivc-Upper  Tier  Theater  Missile  Defense  (UTTMDS)  and  the  Fixed  Distributed 
System  (FDS)“arc  anomalies  in  the  acquisitiOD  process:  deployment  of  UTTMDS  dem/val 
operational  prou) types  was  assessed  by  the  Defense  Acquisition  Board  to  incxease  program 
risk;  and,  FDS  will  be  icrminaied  after  HMD  with  "deployment"  of  one  test  unit  In  addition 
CO  being  too  small  to  be  statistically  significant,  the  sample  selected  is  too  narrowly  focused: 
all  the  programs  considered  include  prototyping- which  precludes  understanding  the  effects  of 
not  prototyping- and  none  have  completed  engineering  and  manufacturing  developmem 
(EMD)-which  precludes  identifying  the  effects  of  dcm/val  prototyping  on  total  EMD  cost  and 
schedule.  In  a  revised  report,  the  IG  staff  should  consider  a  more  broadly  baaed,  statistically 
significant  sample  of  programs. 

ConcUaioH,  Given  the  lack  of  compelling  rationale  supporting  the  current  draft 
repon's  inferred  finding,  it  seems  premature  to  change  DoDI  5000.2  to  mandate  a  highly 
structured,  prescriptive  prototyping  plan,  I  will  be  happy  to  review  the  findings  and 
recommendations  of  a  revised  repon  addressiog  the  issues  raised  in  this  memorandum. 

William  J.  Lynn 
Director 

Program  Analysis  and  Evaluation 


93 


Director,  Defense  Procurement  Comments 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  OErCNSC  PENTAGON 
WASHINGTON  DC  20901-3000 


ACOUMVDON  AND 
rCCHMOUXIV 

DP(DAR) 


W''3.;  1994 


MEMORANDUM  FOR  DIRECTOR,  ACQUISITION  MANAGEMENT  DIRECTORATE, 

OFFICE  OF  THE  INSPECTOR  GENERAL  FOR  AUDITING,  DOD 


THROUGH : 
SUBJECT: 


CHIEF,  CONGRESSIONAL  ACTIONS  AND  INTERNAL  REPORTS 


Draft  Audit  Report,  Project  No,  2AE-0051,  Effectiveness 
of  Prototyping  Acquisition  Strategies  for  Major 
Defense  Acquisition  Programs 


This  responds  to  your  March  30,  1994,  memorandum  requesting 
comments  on  recommendation  3  in  the  subject  draft  report. 

Recommendation  3  recommends  revising  the  Defense  Federal 
Acquisition  Regulation  Supplement  IDFARS)  to  provide  solicitation 
language  to  use  in  requesting  contractor  input,  through  Request 
for  Proposals,  concerning  proposed  prototyping  plans  and  to  allow 
development  of  data  item  descriptions  for  contractor  prototyping 
plans  and  contractor  reports  documenting  the  results  of 
crototyping  risk-reduction  efforts. 

'We  do  not  agree  with  the  recommendation.  Standard 
solicitation  language  would  not  be  useful  because  this  type  of 
solicitation  requirement  should  be  tailored  for  individual 
programs.  We  recommend  instead  revising  the  suggested  DoDI 
5000.2  language  on  prototyping  to  state  that  the  solicitation's 
proposal  preparation  instructions  should  include  a  complete 
description  of  the  prototyping  plan  required  to  be  submitted  with 
the  proposal.  Detailed  comments  are  attached. 

The  recommendation  regarding  development  of  data  item 
descriptions  should  be  directed  to  the  Director  of  the  CALS  Plans 
and  Policy  Division,  as  that  office  has  cognizance  over  data  item 
descriptions . 


Attachment 


Eleanor  R.  Spector 
Director,  Defense  Procurement 


Director,  Defense  Procurement  Conunents 


Final  Report 
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Attachment 


Director  of  Defense  Procurement  Commente 
RecoomendatloD  3 
Draft  Audit  Report 

*Dft  of  Prototypinp  In  Aeguieition  Stretepiee” 
(Project  No.  2AJI-0051) 


DoDlQ  Reepmaendetion  3;  Revise  the  Defense  Federal 

t  aon  Regulation  Supplement  (DFARS)  to  provide  solicitation 
language  to  use  in  reqruesting  contractor  input,  through  Request 
for  rropcsals,  concerning  proposed  prototyping  plans  and  to  allow 
development  of  data  iter,  descriptions  for  contractor  prototyping 
plans  and  contractor  reports  documenting  the  results  of 
prototyping  risk-reduction  efforts. 

DDP  Comment:  Regarding  the  recommendation  to  revise  the 
Zr.\PS  to  provide  the  sclicitation  language  to  use  ir.  reg'jesting 
rcr tract  or  input  concerning  proposed  prototyping  plans,  we 
recc.-nrend  instead  revising  the  suggested  DoDI  500C  2  language  on 
prototyping  to  state  that  the  solicitation's  proposal  preparation 
instructions  should  include  a  complete  description  of  the 
prototyping  plan  required  to  be  submitted  with  the  proposal. 

This  t>Te  of  solicitation  requirement  will  have  to  be  tailored 
fer  each  program,  and  is  not  susceptible  to  standardization.  Any 
•stsroard*  D.-ARS  language  would  become  redundant  *501  lerplate.  " 
v.h;*e  the  real  inf orm»at lor.  about  what  is  required  would  appear  in 
propcsal  preparation  instructions  anyway.  Consequently,  we  also 
recommend  that  the  following  sentence  be  deleted  from  ‘Contractor 
Input  to  Prototyping  Plans’  on  page  29:  ‘To  accomplish  those 
cr:ert:ves  the  (DFARS}  should  be  amended  to  provide  the 
solicitation  language  to  use  in  the  Request  for  Proposals.’ 

Regarding  the  recermenda:  ion  to  revise  the  DFARS  to  allow 
ce  elopment  of  data  ite.-  descriptions  for  contractor  prototyping 
plans  a.nd  contractor  reports,  m>'  organization  does  not  have  a 
rc.e  in  the  authorization  of  data  item  descriptions  (DlDs?  A 
:::  is  originated  by  the  organization  needing  data  delivered 
uncer  a  contract,  when  that  data  cannot  be  obtained  through  use 
of  a.-  existing  DID  listed  in  the  ’Acquisition  Management  Systems 
arc  Data  Requirements  Control  List,’  DoD  S010.12-L.  This 
recc.mmenda t ion  should  be  directed  to  the  CALS  Plans  and  Policy 
Di’*isiorj,  as  that  office  has  cognizance  over  DIDs, 

DoDIG  CoiomeDtt  on  Conpetitlva  Prototyping  by  Subcontract  ora 
(pages  19  through  21) :  These  comments  deal  with  competitive 
prototyping  restricted  to  the  subcontractor  level.  The  authors 
rcr.te.nd  that  'the  winning  subcontractor  must  be  selected  based  on 
the  best  interest  of  the  Government’  and  coim^end  the  Army  BAT 
program's  approach  to  ’control | ling)  the  selection  of  the  winning 
sut rent ractors  .  •  Specifically,  ’(tjo  maintain  control  over 
subcontractor  coirpet  ition.  the  BAT  Program  Of  face ...  inserted 
provisions  into  the  Request  for  Proposals ...  that  allow  the 
Government  to  establish  the  selection  criteria  to  be  used  for 
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choosing  the  subcont rectors ... (and)  provide  chat  no  authorization 
can  be  given  to  proceed  with  a  subcontract  until  the  Government 
has  approved.*  The  authors  view  this  as  a  necessary  element  of 
•effective  management  of  subcontractors*  and  state  their  belief 
that  •similar  contractual  provisions  should  be  used  in  all 
acguisition  programs  where  competitive  prototyping  is  limited  to 
the  subcontractor  level.*  This  discussion  is  not  translated  into 
an  IG  recomnendat ion . 

DDP  Coaaaent;  There  arc  numerous  situations  in  which  the 
Government  has  awarded  a  prime  contract,  and  has  a  critical 
interest  in  the  prime's  post -award  selection  of  a  subcontractor 
for  a  particular  subsystem  or  component.  In  these  cases,  we 
agree  that  the  prime  contract  should  require  Government  approval 
of  the  prime’s  source  selection  plan,  including  selection 
criteria,  before  the  prime  issues  its  solicitation.  Through 
normal  contract  administration,  the  Government  can  ensure  the 
prime  complies  with  the  approved  plan.  We  disagree  that  the 
GoverrjT.ent  should  control  the  subcontract  competition,  or  subject 
the  prime’s  selection  to  Government  approval  or  veto  This  is 
inconsistent  with  holding  the  prime  responsible  for  performance 
of  the  total,  integrated  system.  If  this  degree  of  control  is 
required,  the  subsystem  or  component  should  be  provided  as 
Government -Furni shed  Equipment  after  a  Government -conducted 
source  selection. 

Mi *cel laneoui :  On  page  37,  recommendation  la  (3)  should 

be  rewritten  to  limit  AKPA  review  to  the  prototyping  plans  rather 
tha.n  to  entire  Acquisition  Strategy  Reports 

On  page  42,  footnote  2,  regarding  JPATS  is  incorrect  There 
IS  nc  JPATS  contract.  The  Air  Force  has  not  yet  issued  the 

Request  for  Proposals  intended  to  lead  to  the  first  JPATS  50 

contract . 
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